Diagnosing the Environment Bias in Vision-and-Language Navigation
Open accessAbstract
Vision-and-Language Navigation (VLN) requires an agent to follow natural-language instructions, explore the given environments, and reach the desired target locations. These step-by-step navigational instructions are crucial when the agent is navigating new environments about which it has no prior knowledge. Most recent works that study VLN observe a significant performance drop when tested on unseen environments (i.e., environments not used in training), indicating that the neural agent models are highly biased towards training environments. Although this issue is considered as one of the major challenges in VLN research, it is still under-studied and needs a clearer explanation. In this work, we design novel diagnosis experiments via environment re-splitting and feature replacement, looking into possible reasons for this environment bias. We observe that neither the language nor the underlying navigational graph, but the low-level visual appearance conveyed by ResNet features directly affects the agent model and contributes to this environment bias in results. According to this observation, we explore several kinds of semantic representations that contain less low-level visual information, hence the agent learned with these features could be better generalized to unseen testing environments. Without modifying the baseline agent model and its training method, our explored semantic features significantly decrease the performance gaps between seen and unseen on multiple datasets (i.e. R2R, R4R, and CVDN) and achieve competitive unseen results to previous state-of-the-art models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The process toward large-scale training involves key sub-problems such as building navigation graphs, recovering corrupted rendered images, and generating navigational instructions, which significantly influence the quality of collected data.
“the process towards such large-scale training involves solving a series of key sub-problems such as how to build navigation graphs [10, 30, 37], how to recover corrupted rendered images [8, 43], and how to generate navigational instructions [23, 25, 78, 83], which significantly influence the quality of collected data”
From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [83]
Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · see citing works
Matterport3D: Learning from RGB-D Data in Indoor Environments
Chang, A., Dai, A., Funkhouser, T., Halber, M., Niebner, M., Savva, M., Song, S., Zeng, A., & Zhang, Y. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00081 · Cited by 1,858 · see citing works
Self-citationLearning to Navigate Unseen Environments: Back Translation with Environmental Dropout
Tan, H., Yu, L., & Bansal, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1268 · Cited by 293 · see citing works
Self-citationSpeaker-Follower Models for Vision-and-Language Navigation
Fried, D., Hu, R., Cirik, V., Rohrbach, A., Andreas, J., Morency, L.-P., Berg-Kirkpatrick, T., Saenko, K., Klein, D., & Darrell, T. (2018) · arXiv · doi:10.48550/arxiv.1806.02724 · Cited by 291 · see citing works
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
Ma, C.-Y., Lu, J., Wu, Z., AlRegib, G., Kira, Z., Socher, R., & Xiong, C. (2019) · arXiv · doi:10.48550/arxiv.1901.03035 · Cited by 200 · see citing works
Transferable Representation Learning in Vision-and-Language Navigation
Huang, H., Jain, V., Mehta, H., Ku, A., Magalhaes, G., Baldridge, J., & Ie, E. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00750 · Cited by 82 · see citing works
Flagged first, then by citations
Research works printed but not linked: 27
- Anderson et al., 2018a] P. Anderson, X. He, et al. Bottom- up and top-down attention for image captioning and visual question answering. In CVPR, 2018.
- Anderson et al., 2018b] P. Anderson, Q. Wu, et al. Vision- and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, 2018.
- P. Anderson, A. Shrivastava, et al. Chasing ghosts: Instruction following as bayesian state tracking. In NeurIPS, 2019.
- D. Bahdanau, K. Cho, and Y. Ben- gio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
- G. Blanchard, G. Lee, and C. Scott. Generalizing from several related classification tasks to a new unlabeled sample. In NeurIPS, 2011.
- V. Blukis, D. Misra, et al. Mapping navigation instructions to continuous control actions with position-visitation prediction. In CoRL, 2018.
- F. M. Carlucci, A. D'Innocente, et al. Domain generalization by solving jigsaw puzzles. In CVPR, 2019.
- Chen et al., 2019a] C. Chen, Z. Chen, et al. Joint domain alignment and discriminative feature learning for unsuper- vised deep domain adaptation. In AAAI, 2019.
- Chen et al., 2019b] H. Chen, A. Suhr, et al. Touchdown: Natural language navigation and spatial reasoning in vi- sual street environments. In CVPR, 2019.
- A. Das, S. Datta, et al. Embodied question answering. In CVPR, 2018.
- Goodfellow, J. Pouget-Abadie, et al. Generative adversarial nets. In NeurIPS, 2014.
- R. Hu, D. Fried, et al. Are you looking? grounding to multiple modalities in vision-and-language navigation. In ACL, 2019.
- V. Jain, G. Magalhaes, et al. Stay on the path: Instruction fidelity in vision-and-language naviga- tion. In ACL, 2019.
- L. Ke, X. Li, et al. Tactical rewind: Self- correction via backtracking in vision-and-language navi- gation. In CVPR, 2019.
- C. Lin. Rouge: A package for automatic evalu- ation of summaries. In Text summarization branches out, pages 74-81, 2004.
- M. Long, Z. Cao, et al. Conditional ad- versarial domain adaptation. In NeurIPS, 2018.
- Ma et al., 2019b] C. Ma, Z. Wu, et al. The regretful agent: Heuristic-aided navigation through progress estimation. In CVPR, 2019.
- M. MacMahon, B. Stankiewicz, and B. Kuipers. Walk the talk: connecting language, knowledge, and action in route instructions. In AAAI, 2006.
- P. Mirowski, M. Grimes, et al. Learning to navigate in cities without a map. In NeurIPS, 2018.
- K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representation. In ICML, 2013.
- K. Papineni, S. Roukos, et al. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
- A. Rozantsev, M. Salzmann, and P. Fua. Residual parameter transfer for deep domain adap- tation. In CVPR, 2018.
- O. Russakovsky, J. Deng, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-252, 2015.
- Thomason et al., 2019b] J. Thomason, M. Murray, et al. Vision-and-dialog navigation. In CoRL, 2019.
- Wang et al., 2018] X. Wang, W. Xiong, et al. Look before you leap: Bridging model-free and model-based reinforce- ment learning for planned-ahead vision-and-language nav- igation. In ECCV, 2018.
- Wang et al., 2019] X. Wang, Q. Huang, et al. Reinforced cross-modal matching and self-supervised imitation learn- ing for vision-language navigation. In CVPR, 2019.
- L. Yu, X. Chen, et al. Multi-target embod- ied question answering. In CVPR, 2019.
Cited by
Sorted by citations
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
Airbert: In-domain Pretraining for Vision-and-Language Navigation
Guhur, P.-L., Tapaswi, M., Chen, S., Laptev, I., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00166 · Cited by 139 · see citing works
Envedit: Environment Editing for Vision-and-Language Navigation
Li, J., Tan, H., & Bansal, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01497 · Cited by 87 · see citing works
Self-citationScaling Data Generation in Vision-and-Language Navigation
Wang, Z., Li, J., Hong, Y., Wang, Y., Wu, Q., Bansal, M., Gould, S., Tan, H., & Qiao, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01103 · Cited by 64 · see citing works
“As suggested in MARVAL [37], we also experiment with applying a more powerful visual encoder, CLIP ViT-H/14 [64], and the image augmentation method EnvEdit [47] to our approach, leading to a remarkable 80% SR, and reducing the longlasting generalization gap between seen and unseen environments [84] to less than 1%.”
MethodSelf-citation
- Authors
- Zhang, Y., Tan, H., & Bansal, M.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- Venue
- Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence
- Published
- 2020-07
- Pages
- 890-897
- OpenAlex
- W3034984114
- Cited by
- 51
- References
- 8
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 7 · 2023: 9 · 2022: 10 · 2021: 11 · 2020: 4
- FWCI
- 2.59
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Institutes of Health (1r01lm013329-01)
- Office of Naval Research (N000141812871, N00014-18, N00014)
- Army Research Office (W911NF)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Automotive Engineering › Spatial Cognition and Navigation
- Keywords
- vision-and-language navigation, generalization to unseen environments, semantic feature representation, natural language instructions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
