Cited by 88Multimodal Machine Learning ApplicationsReinforcement Learning in RoboticsSpatial Cognition and Navigation

Envedit: Environment Editing for Vision-and-Language Navigation

Open access
Li, Jialu, Tan, Hao, Bansal, Mohit·2022·2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Abstract

In Vision-and-Language Navigation (VLN), an agent needs to navigate through the environment based on nat-ural language instructions. Due to limited available data for agent training and finite diversity in navigation environments, it is challenging for the agent to generalize to new, unseen environments. To address this problem, we propose Envedit, a data augmentation method that cre-ates new environments by editing existing environments, which are used to train a more generalizable agent. Our augmented environments can differ from the seen environ-ments in three diverse aspects: style, object appearance, and object classes. Training on these edit-augmented environments prevents the agent from overfitting to existing en-vironments and helps generalize better to new, unseen en-vironments. Empirically, on both the Room-to-Room and the multi-lingual Room-Across-Room datasets, we show that our proposed Envedit method gets significant im-provements in all metrics on both pre-trained and non-pre-trained VLN agents, and achieves the new state-of-the-art on the test leaderboard. We further ensemble the VLN agents augmented on different edited environments and show that these edit methods are complementary.11Code and data are available at https://github.com/jialuli-luka/EnvEdit.

Integrity signals

Registry

Crossref · 10.1109/cvpr52688.2022.01497

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • KitchenShift is useful for benchmarking imitation learning and reinforcement learning but does not support real-world reproducibility; its domain shift testing provides insight into real-world uncertainty modeling.

    “KitchenShift [292] is particularly useful for benchmarking imitation learning (IL) and reinforcement learn-ing (RL) strategies, but like RLBench [288], it does not support real-world reproducibility, though its domain shift testing provides insight into real-world uncertainty modeling.”
  • Isaac Sim provides high-fidelity physics, is ideal for reinforcement learning, imitation learning, and robotic vision, and is used in KitchenShift and GenSim2.

    “It provides high-fidelity physics simulation, making it ideal for reinforcement learning (RL), imitation learning (IL), and robotic vision tasks, and is widely used in KitchenShift [292] and GenSim2 [295] benchmarks.”
  • KitchenShift focuses on robot generalization under environmental domain shifts in a simulated kitchen, introducing shifts in object materials, lighting, and camera angles to evaluate policy robustness.

    “In contrast, KitchenShift [292] focuses on robot generalization under environmental domain shifts, simulating a kitchen environment where the robot interacts with various objects, such as opening an oven, moving utensils, and mixing liquids.”
  • Applying CLIP ViT-H/14 and EnvEdit to our approach leads to 80% SR.

    “we also experiment with applying a more powerful visual encoder, CLIP ViT-H/14 [64], and the image augmentation method EnvEdit [47] to our approach, leading to a remarkable 80% SR”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [47]

Cites

Flagged first, then by citations

4 of the 72 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Li, J., Tan, H., & Bansal, M.
Affiliations
University of North Carolina at Chapel Hill (US)
Venue
2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2022-06
Pages
15386-15396
ISSN
1063-6919 (ISSN-L)
OpenAlex
W4312434279
Cited by
87
References
72
Citations by year
2026: 14 · 2025: 26 · 2024: 26 · 2023: 16 · 2022: 5
FWCI
6.62
Type
conference paper
Language
en
Open access
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1846185, CAREER)
  • Defense Advanced Research Projects Agency (FA8750-19-2-1004, FA8750)
  • Office of Naval Research (N000141812871)
  • Army Research Office (W911NF2110220)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Automotive Engineering › Spatial Cognition and Navigation
Keywords
vision-and-language navigation, data augmentation, generalization to unseen environments, overfitting

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review