The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigation
Open accessAbstract
Vision-and-Language Navigation (VLN) requires an agent to find a path to a remote location on the basis of natural-language instructions and a set of photo-realistic panoramas. Most existing methods take the words in the instructions and the discrete views of each panorama as the minimal unit of encoding. However, this requires a model to match different nouns (e.g., TV, table) against the same input view feature. In this work, we propose an object-informed sequential BERT to encode visual perceptions and linguistic instructions at the same fine-grained level, namely objects and words. Our sequential BERT also enables the visual-textual clues to be interpreted in light of the temporal context, which is crucial to multi-round VLN tasks. Additionally, we enable the model to identify the relative direction (e.g., left/right/front/back) of each navigable location and the room type (e.g., bedroom, kitchen) of its current and final navigation goal, as such information is widely mentioned in instructions implying the desired next and final locations. We thus enable the model to know-where the objects lie in the images, and to know-where they stand in the scene. Extensive experiments demonstrate the effectiveness compared against several state-of-the-art methods on three indoor VLN tasks: REVERIE, NDH, and R2R. Project repository: https://github.com/YuankaiQi/ORIST
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A simple LSTM-based model (EnvDrop) trained from scratch results in a higher Bleu-4 score than a fine-tuned GPT-2.
“a simple LSTMbased model (EnvDrop [74]) trained from-scratch results in a higher Bleu-4 score [57] than a fine-tuned GPT-2 [65].”
From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [57]
Vanilla Transformers can be understood from a geometrically topological perspective because self-attention can model each tokenized input from any modality as a fully-connected graph in topological geometry space.
“We highlight that Vanilla Transformers can be understood from a geometrically topological perspective [157], because due to the self-attention mechanism, given each tokenized input from any modalities, Vanilla self-attention (Transformer) can model it as a fully-connected graph in topological geometry space [158].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [157]
The citation lists haven’t been fetched yet
References 43 · Cited by 80
- Authors
- Qi, Y., Pan, Z., Hong, Y., Yang, M.-H., van den Hengel, A., & Wu, Q.
- Affiliations
- The University of Adelaide (AU)
- Monash University (AU)
- Australian National University (AU)
- Google (United States) (US)
- The University of Adelaide,Australian Institute for Machine Learning
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 1635-1644
- OpenAlex
- W3205276578
- Cited by
- 80
- References
- 43
- Citations by year
- 2026: 8 · 2025: 8 · 2024: 21 · 2023: 25 · 2022: 15 · 2021: 3
- FWCI
- 5.22
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1149783, CAREER)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Automotive Engineering › Spatial Cognition and Navigation
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- vision-and-language navigation, indoor navigation, temporal context, natural language instructions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
