CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments
Open accessAbstract
Audio-visual navigation of an agent towards locating an audio goal is a challenging task especially when the audio is sporadic or the environment is noisy. In this paper, we present CAVEN, a Conversation-based Audio-Visual Embodied Navigation framework in which the agent may interact with a human/oracle for solving the task of navigating to an audio goal. Specifically, CAVEN is modeled as a budget-aware partially observable semi-Markov decision process that implicitly learns the uncertainty in the audio-based navigation policy to decide when and how the agent may interact with the oracle. Our CAVEN agent can engage in fully-bidirectional natural language conversations by producing relevant questions and interpret free-form, potentially noisy responses from the oracle based on the audio-visual context. To enable such a capability, CAVEN is equipped with: i) a trajectory forecasting network that is grounded in audio-visual cues to produce a potential trajectory to the estimated goal, and (ii) a natural language based question generation and reasoning network to pose an interactive question to the oracle or interpret the oracle's response to produce navigation instructions. To train the interactive modules, we present a large scale dataset: AVN-Instruct, based on the Landmark-RxR dataset. To substantiate the usefulness of conversations, we present experiments on the benchmark audio-goal task using the SoundSpaces simulator under various noisy settings. Our results reveal that our fully-conversational approach leads to nearly an order-of-magnitude improvement in success rate, especially in localizing new sound sources and against methods that use only uni-directional interaction.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
3 of 3 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured
- Findingown data
The question policy is chosen more frequently when the agent has moderate confidence.
“It is note-worthy that the question policy is invoked more often when the agent is moderately confident.”
Quote in the paperNot a substantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Rests on an assumption: The AVN-Instruct dataset generated by the pre-trained QuestionNet is valid training data for the language interaction modules.
- Findingown data
In the authors' experiments, omitting the penalty for correct questions improved results, with δ_ques set to zero.
“In our experiments, we find that not penalizing the agent for correct questions leads to better results, i.e., δ ques (m) = 0.”
Quote in the paperSubstantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Findingown data
CAVEN performed better than Paul et al. even when the oracle was noisy.
“Even with a noisy oracle, i.e., Noisy Language in Table 1, we achieve better performances compared to Paul et al.”
Quote in the paperSubstantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Rests on an assumption: The AVN-Instruct dataset generated by the pre-trained QuestionNet is valid training data for the language interaction modules.
- Rests on an assumption: The simulated noisy oracle, created by random trajectories or 25% yes/no flips, adequately represents real-world noise.
- Findingown data
In the distractor-sound setting, CAVEN achieved gains of 5.5% on SPL and 3.1% on SNO relative to the closest competitor.
“Even under such challenging circumstances, as shown in Table 2, we notice a 5.5% and a 3.1% gain on SPL and SNO, respectively against our closest competitor.”
Quote in the paperSubstantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Rests on an assumption: The AVN-Instruct dataset generated by the pre-trained QuestionNet is valid training data for the language interaction modules.
- Findingown data
CAVEN achieved a 12% improvement on the SNO metric compared to Paul et al., the closest competitor, in both heard and unheard conditions.
“CAVEN exhibits a 12% gain on the newly introduced SNO metric over Paul et al., our closest competitor, in both heard and unheard cases.”
Quote in the paperSubstantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Rests on an assumption: The AVN-Instruct dataset generated by the pre-trained QuestionNet is valid training data for the language interaction modules.
- Findingown data
CAVEN's experiments on semantic audio-goal navigation in SoundSpaces under challenging scenarios with intermittent sounds show a gain of nearly 12% over competing approaches on success rate.
“Our experimental results using the above setup clearly bring out the benefits of enabling the agent to converse with the oracle, demonstrating a solid gain of nearly 12% over competing approaches on the success rate.”
Quote in the paperSubstantial claim - Findingown data
CAVEN's fully-conversational approach achieves nearly an order-of-magnitude improvement in success rate, especially in localizing new sound sources and against methods using only uni-directional interaction.
“Our results reveal that our fully-conversational approach leads to nearly an order-of-magnitude improvement in success rate, especially in localizing new sound sources and against methods that use only uni-directional interaction.”
- Also stated in: § Introduction · boosted
“Our experiments demonstrate state-of-the-art performances against related prior approaches by an order-ofmagnitude increase in success rate.”
Quote in the paperSubstantial claim - Also stated in: § Introduction · boosted
- Findingown data
The full CAVEN model with language capabilities showed significant gains on every metric.
“The table shows that our full model -CAVEN (language), achieves significant improvements across all metrics.”
Quote in the paperSubstantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Rests on an assumption: The AVN-Instruct dataset generated by the pre-trained QuestionNet is valid training data for the language interaction modules.
- Findingown data
AVN-Instruct is a large-scale audio-visual-language navigation subinstruction dataset based on Landmark-RxR, containing 41.5k pairs of audio-goal, trajectory, and language instructions.
“To train the interactive modules, we present a large scale dataset: AVN-Instruct, based on the Landmark-RxR dataset.”
- Also stated in: § Introduction · plain
“we introduce AVN-Instruct -a novel audio-visual-language navigation subinstruction dataset with 41.5k pairs of audio-goal, trajectory, and language instructions.”
Quote in the paperSubstantial claim - Also stated in: § Introduction · plain
- Interpretationown data
This result demonstrates the robustness of the proposed framework.
“showing the robustness of our framework.”
Quote in the paperSubstantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Rests on an assumption: The AVN-Instruct dataset generated by the pre-trained QuestionNet is valid training data for the language interaction modules.
- Rests on an assumption: The simulated noisy oracle, created by random trajectories or 25% yes/no flips, adequately represents real-world noise.
- Interpretationnone given
The paper proposes new metrics SNO and SNI for evaluating language-guided navigation tasks.
“We propose a novel audio-visual-language navigation subinstruction dataset, AVN-Instruct to pre-train embodied navigation models along with new metrics to evaluate language-guided navigation tasks, dubbed SNO and SNI.”
Quote in the paperSubstantial claim - Interpretationreasoning
CAVEN can engage in fully bidirectional natural language conversations with an oracle, producing questions and interpreting free-form, potentially noisy responses based on audio-visual context.
“Our CAVEN agent can engage in fully-bidirectional natural language conversations by producing relevant questions and interpret free-form, potentially noisy responses from the oracle based on the audio-visual context.”
- Also stated in: § Introduction · plain
“We present CAVEN, a multimodal navigation agent that is, for the first time, capable of fully-bidirectional interaction with an oracle in free-form natural language, thereby facilitating easy communication.”
Quote in the paperSubstantial claim - Also stated in: § Introduction · plain
- Interpretationnone given
CAVEN is the first multimodal navigation agent capable of fully-bidirectional interaction with an oracle in free-form natural language.
“We present CAVEN, a multimodal navigation agent that is, for the first time, capable of fully-bidirectional interaction with an oracle in free-form natural language, thereby facilitating easy communication.”
Quote in the paperNot a substantial claim - Interpretationnone given
CAVEN is modeled as a budget-aware partially observable semi-Markov decision process that learns uncertainty in the audio-based navigation policy to decide when and how the agent interacts with the oracle.
“Specifically, CAVEN is modeled as a budget-aware partially observable semi-Markov decision process that implicitly learns the uncertainty in the audio-based navigation policy to decide when and how the agent may interact with the oracle.”
Quote in the paperSubstantial claim - Interpretationnone given
CAVEN uses a novel budget-aware RL policy that integrates the question module as an additional navigation policy to existing ones.
“We design a novel budget-aware RL policy, which integrates the question module as an additional navigation policy to existing ones.”
Quote in the paperSubstantial claim - Interpretationown data
The results indicate that the agent gains substantially more from the two novel language components.
“This clearly shows that the agent benefits much more from both our novel language components.”
Quote in the paperSubstantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Rests on an assumption: The AVN-Instruct dataset generated by the pre-trained QuestionNet is valid training data for the language interaction modules.
- Interpretationnone given
CAVEN can seek occasional human or oracle help at any location.
“Contrary to these methods, we empower our CAVEN agent with: (i) the ability to seek occasional human/oracle help at any location and (ii) competence for natural language-based scene grounded conversations with an oracle for effective navigation.”
Quote in the paperSubstantial claim - Interpretationreasoning
The higher success rate suggests that rewarding correct questions significantly influences the learning of trajectory forecasting.
“The success rate is higher suggesting that the incentive the agent receives in making a correct question influences learning of the trajectory forecasting significantly.”
Quote in the paperSubstantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Rests on an assumption: The AVN-Instruct dataset generated by the pre-trained QuestionNet is valid training data for the language interaction modules.
- Interpretationreasoning
In the distractor-sound setting, dialogue between the agent and oracle becomes even more important.
“Under this setting, the conversations between the agent and oracle becomes even more critical.”
Quote in the paperSubstantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Interpretationown data
Distractor sounds negatively impact audio-goal estimation, increasing uncertainty in the agent's decisions.
“The presence of distractor sounds adversely affects the estimation of the audio-goal, which results in more uncertainty in the agent's decisionmaking.”
Quote in the paperSubstantial claim - Interpretationown data
The differential reward scheme implicitly encourages the agent to learn correct trajectories toward the audio goal, thereby improving performance.
“Such a differential reward implicitly reinforces the agent to learn correct trajectories to the audio goal, improving performance.”
Quote in the paperSubstantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Interpretationown data
The difference between the two penalties incentivizes the agent to make more correct trajectory predictions than when the penalty gap is smaller.
“Thus, the gap between the two penalties act as an incentive for the agent to make more number of correct trajectory predictions than in a case where this penalty gap is lower (e.g., δ ques = 0.5, 1.0 in which case it is similar to the penalty it receives for the wrong question).”
Quote in the paperSubstantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Interpretationcitation
CAVEN can use audio-visual cues for navigation.
“Specifically, CAVEN can either use the audio-visual cues for its navigation (as in prior works (Chen et al. 2020(Chen et al. , 2021a;;Gan et al. 2020)) or in case the agent is uncertain of which navigation step to take, it can interact with the oracle in two distinct modes:”
Quote in the paperSubstantial claim
- May combine two claims. 12
- The quote only partly states this claim. 1
- Classified as probably not a substantial claim. 4
May combine two claims.
PremisecitationThe oracle in AVLEN has access to the 3D space of the full environment and can provide plausible instructions for navigation.
“Note that the oracle in AVLEN has access to the 3D space of the full environment and thus can provide plausible instructions for navigation, however CAVEN has only a very restricted view of the scene in its vicinity, thus making this task of creating a question at the agent's side entirely different from that of the oracle's.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Paul et al. 2022 source
The Proposed Method (pp. 4, 7) says the oracle knows the shortest path from the agent to the AudioGoal and generates a short natural-language instruction from a segment of that path. This supports its ability to give navigation instructions, but our copy does not establish that it has access to the full environment’s 3D space, rather than the path and observations used to generate the instruction. (LLM, conf=0.82)
Experimental — accuracy not yet measured
May combine two claims.
PremisecitationThe AVLEN agent could ask only a fixed question while the oracle could provide a natural-language reply that guided the agent toward the goal.
“In particular, the AVLEN agent could ask only a fixed question (e.g.,”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Paul et al., 2022 source
May combine two claims.
PremisecitationThe AVLEN agent cannot pose questions, which burdens the oracle with charting a path whenever help is sought.
“However, there is no provision of posing questions, which burdens the oracle with the task of chalking out a path to the goal whenever help is sought.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Paul et al., 2022 source
The Problem Formulation (p. 4) gives the agent a query action, but describes no facility for posing a specific question; the oracle responds with a navigation instruction. The Problem Setup (p. 4) says that instruction describes only an initial segment of the shortest path, not a complete path to the goal. Thus the claim captures the limited form of help-seeking but overstates what the oracle must provide. (LLM, conf=0.82)
Experimental — accuracy not yet measured
The quote only partly states this claim.
Findingown dataThe learned selector policy π_s outperformed the alternative strategies tested.
“Our results show learning of π s is better.”
Quote in the paperSubstantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Rests on an assumption: The AVN-Instruct dataset generated by the pre-trained QuestionNet is valid training data for the language interaction modules.
May combine two claims.
Findingown dataQuantitative evaluations across various noisy settings showed large performance improvements over competing methods, supporting the benefits of the proposed interaction policies and architecture.
“Quantitative evaluations of CAVEN under various noisy problem settings, using established and novel metrics, demonstrate large improvements in performance over competing methods, substantiating the benefits of our proposed interaction policies and our architecture.”
Quote in the paperSubstantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Rests on an assumption: The AVN-Instruct dataset generated by the pre-trained QuestionNet is valid training data for the language interaction modules.
May combine two claims.
Findingown dataUnder the oracle instruction budget, CAVEN asked a correct question roughly 40% of the time, which reduced its penalty.
“Given the budget on directly receiving instructions from the oracle, we find that CAVEN poses a correct question about 40% of the time, thereby incurring less penalty.”
Quote in the paperNot a substantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Rests on an assumption: The AVN-Instruct dataset generated by the pre-trained QuestionNet is valid training data for the language interaction modules.
May combine two claims.
Findingown dataThe audio-goal policy is selected when the agent has high confidence, while the language-based policy is selected when confidence is low.
“As shown in Figure 4, the audio-goal is invoked when the agent is highly confident and the language-based policy is invoked when agent's confidence is low.”
Quote in the paperNot a substantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Rests on an assumption: The AVN-Instruct dataset generated by the pre-trained QuestionNet is valid training data for the language interaction modules.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationown dataThe newly introduced SNI and SNO metrics help clarify the performance improvement observed in conversational settings.
“These additional metrics help explain the performance gain under conversational settings.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningCAVEN addresses all six challenges (when to use language, what interaction type, how to phrase questions, how to make the oracle understand, how the oracle responds, how frequently to ask) within a single framework using a budget-aware POSMDP and reinforcement learning with novel rewards.
“However in CAVEN, we tackle all these challenges within a single framework, by proposing a novel budget-aware partially observable semi-Markov decision process (POSMDP), using a reinforcement learning framework by introducing novel learning rewards.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
InterpretationreasoningCAVEN's POSMDP disentangles interactive audio goal navigation into three low-level options: audio-visual navigation, instruction-guided navigation, and bi-directional question-answer navigation.
“In our setup, we disentangle agents' interactive audio goal navigation process into three low level temporal abstractions (i.e., options): i) audio-visual navigation ξ g , ii) instruction-guided navigation ξ l , iii) bi-directional questionanswer navigation ξ ques .”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationnone givenCAVEN's question module consists of a trajectory forecasting network grounded in audio-visual cues, a natural language question generation and reasoning network, and a question decoder (FollowerNet) on the oracle.
“CAVEN is equipped with: i) a trajectory forecasting network that is grounded in audio-visual cues to produce a potential trajectory to the estimated goal, and (ii) a natural language based question generation and reasoning network to pose an interactive question to the oracle or interpret the oracle's response to produce navigation instructions.”
- Also stated in: § Introduction · plain
“We introduce a novel question module for bi-directional interaction with the oracle consisting of: (i) a trajectory forecasting module grounded on both visual scenes and audio cues, (ii) a question generation module, and (iii) a question decoder (FollowerNet, on the oracle).”
Quote in the paperSubstantial claim- Also stated in: § Introduction · plain
May combine two claims.
InterpretationreasoningCAVEN sees only a narrow local region of the scene, so generating a question on the agent side differs from the oracle's task.
“Note that the oracle in AVLEN has access to the 3D space of the full environment and thus can provide plausible instructions for navigation, however CAVEN has only a very restricted view of the scene in its vicinity, thus making this task of creating a question at the agent's side entirely different from that of the oracle's.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningAlthough asking questions risks penalties for wrong questions, the question policy benefits from using the agent's own beliefs to seek confirmation and reduce uncertainty, leading to efficient navigation.
“Though it potentially risks being penalized by asking wrong questions, it benefits from seeking confirmation from the oracle using its own beliefs to alleviate uncertainty, thus navigating efficiently.”
Quote in the paperSubstantial claim- Rests on an assumption: The oracle supplies useful language instructions when the agent queries it.
- Rests on an assumption: The AVN-Instruct dataset generated by the pre-trained QuestionNet is valid training data for the language interaction modules.
Probably not a substantial claim: may not pass S4 (based).
Interpretationnone givenCAVEN is robust to noisy feedback from the oracle.
“Further, our agent is also robust to noisy feedback from oracle.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningWhen uncertain, CAVEN can interact with the oracle in a question mode, forecasting a trajectory, framing a question, and interpreting the response.
“(i) a question mode, in which the agent forecasts a plausible trajectory based on audio-goal belief, using which it frames a natural language question to be posed to the oracle, and subsequently interpreting the oracle's response to the question”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationnone givenAsking correct questions results in minimal oracle effort in producing a response.
“Since, asking correct question results in minimal oracle effort in producing a response, we define a dynamic penalty based on the question by”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationnone givenCAVEN is a conversation-based audio-visual embodied navigation framework where an agent can interact with a human or oracle to navigate to an audio goal.
“In this paper, we present CAVEN, a Conversation-based Audio-Visual Embodied Navigation framework in which the agent may interact with a human/oracle for solving the task of navigating to an audio goal.”
Quote in the paperSubstantial claim
The authors state this as an assumption.
The three options are valid in any state of the environment.
“These options are assumed valid in any state of the environment, i.e., S ξg , S ξ l , S ξques ∈ S.”
The authors state this as an assumption.
The sound is semantically unique and produced by a static object, though it may be noisy, sporadic, or mixed with other environmental sounds.
“As in (Chen et al. 2020), we assume the sound is semantically unique and is produced by a static object, however the audio could be noisy, sporadic, or mixed up with other environmental sounds.”
The authors state this as an assumption.
After receiving a question, the oracle gives a yes response if its interpretation of the question in its own state space yields actions matching its estimate of the actions along the ground-truth geodesic to the goal; otherwise it responds no followed by a short guiding sentence.
“Further, we assume the oracle after receiving this question, will either give a”
The authors state this as an assumption.
The number of times an agent can receive direct navigation instructions from the oracle is limited by a budget.
“In our new problem setup, we also assume that the number of times an agent can receive direct navigation instructions from the oracle (as a result of a wrong question or when it directly queries) is limited by a budget so that the agent only seeks help when necessary.”
The paper relies on this without support.
The oracle supplies useful language instructions when the agent queries it.
“When invoked, the agent receives instructions, similar to when a wrong question is posed to the oracle.”
“To cater to such cases, we equip the agent to directly query the oracle for language-based instructions.”
The paper relies on this without support.
The AVN-Instruct dataset generated by the pre-trained QuestionNet is valid training data for the language interaction modules.
“Then we adopt the pre-trained QuestionNet to synthesize a dataset called AVN-Instruct, which contains a total of 41.5k dense pairs of sub-instructions, audio-goal, and visual scene under the state space of Soundspace simulator.”
“Before integrating the modules into the RL framework, we fine-tune the whole question module end-to-end on AVN-Instruct with a set of 500 and 1000 samples for validation and testing.”
The paper relies on this without support.
The simulated noisy oracle, created by random trajectories or 25% yes/no flips, adequately represents real-world noise.
“To induce noise, we either ground the generated oracle's instructions on random trajectories or switch 'yes'/'no' responses, both with a chance of 25%.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Correct questions can still incur a frequency penalty
MinorThe text says a correct question incurs no penalty, but the total question penalty also includes ζfques for asking too many questions. That term can apply even when the question is correct.
“and no penalty will incur when E(q) = 1.”
“ζ ques (m) = ζ l (m, K ′ ) δ ques (m) + ζ fques (m), (2) where ζ fques is the penalty for asking questions too many times”
A tested reward value lies outside its stated range
MinorThe definition restricts the parameter for a non-'no' response to values below 1, yet the paper discusses and reports an experiment with that parameter set to 1.0.
“else δ ques (m) ∈ [0, 1) is a constant.”
“δ ques = 0.5, 1.0 in which case it is similar to the penalty it receives for the wrong question”
Success-rate improvement is described at incompatible scales
MinorThe paper characterizes its success-rate gain over competing approaches as nearly 12%, but also calls the gain over related approaches an order of magnitude. Those descriptions do not agree for the reported comparison.
“Our experiments demonstrate state-of-the-art performances against related prior approaches by an order-ofmagnitude increase in success rate.”
“Our experimental results using the above setup clearly bring out the benefits of enabling the agent to converse with the oracle, demonstrating a solid gain of nearly 12% over competing approaches on the success rate.”
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1× · see citing works
“…these views are 512D, obtained by projecting the concatenated 2048D RGB image features encoded via a ResNet-152 network pretrained on the ImageNet dataset (He et al. 2016) and the 64D egocentric occupancy map features encoded through the pretrained 3-layer CNN encoder from the TrajectoryNet module.”
MethodVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 6× · see citing works
“The output of the CNN is then projected to 128D using a Multilayer Perceptron (MLP), before being added to a standard sinusoidal position encoding, as is common practice (Vaswani et al. 2017).”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 1× · see citing works
“Recent approaches have begun using transformer-based architectures, such as BERT (Devlin et al. 2018) for VLN (Hong et al. 2021; Majumdar et al. 2020).”
MethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 4× · see citing works
“…ResNet-152 features from g view , coupled with the position encoding of g sub , and the embeddings of hitherto produced words in the question (e.g., GloVe (Pennington, Socher, and Manning 2014) or CLIP (Radford et al. 2021)), and proceeds to generate the next word in the question autoregressively.”
BackgroundMethodInfluentialBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2× · see citing works
“Ablations on QuestionNet To prove the capability of pre-trained navigation language model on the 150k Landmark-RxR sub-instructions dataset (in MatterPort3D simulator) on our QuestionNet, we evaluate our generated questions or instructions using metrics commonly used for assessing the quality of language translation o…”
MethodLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 4× · see citing works
“…ResNet-152 features from g view , coupled with the position encoding of g sub , and the embeddings of hitherto produced words in the question (e.g., GloVe (Pennington, Socher, and Manning 2014) or CLIP (Radford et al. 2021)), and proceeds to generate the next word in the question autoregressively.”
BackgroundMethodInfluentialCIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 2× · see citing works
“Ablations on QuestionNet To prove the capability of pre-trained navigation language model on the 150k Landmark-RxR sub-instructions dataset (in MatterPort3D simulator) on our QuestionNet, we evaluate our generated questions or instructions using metrics commonly used for assessing the quality of language translation o…”
MethodLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1× · see citing works
“The spark of recent advancements in large language models (LLMs) (Bubeck et al. 2023; Touvron et al. 2023; OpenAI 2023) has brought along new opportunities in improving multi-modal robot navigation.”
BackgroundTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119 · cited 1× · see citing works
“Putting it all together, our objective to learn these policies Here, b ′ is the updated belief and Z ′ is the multi-time transition function (Sutton, Precup, and Singh 1999) given by: Below, we detail the architecture of each of these policies.”
BackgroundTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1× · see citing works
“In NavGPT, (Zhou et al. 2023) the possibility of integrating ChatGPT (Ouyang et al. 2022) with a vision foundation model: BLIP-2 (Li et al. 2023) into its prompting setup to perform multi-modal reasoning to navigate in a zero-shot manner was explored.”
MethodPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1× · see citing works
“…deep neural networks and sophisticated language models have led to significant advancements in building conversational agents that can collaborate with humans in solving challenging reasoning tasks (Peng et al. 2023; Ram et al. 2018; Chowdhery et al. 2022; Gupta and Kembhavi 2023; You et al. 2022).”
BackgroundMatterport3D: Learning from RGB-D Data in Indoor Environments
Chang, A., Dai, A., Funkhouser, T., Halber, M., Niebner, M., Savva, M., Song, S., Zeng, A., & Zhang, Y. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00081 · Cited by 1,858 · cited 2× · see citing works
“It uses Matter-Port3D environment scans (Chang et al. 2017).”
MethodSparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 1× · see citing works
“The spark of recent advancements in large language models (LLMs) (Bubeck et al. 2023; Touvron et al. 2023; OpenAI 2023) has brought along new opportunities in improving multi-modal robot navigation.”
BackgroundHabitat: A Platform for Embodied AI Research
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00943 · Cited by 1,237 · cited 1× · see citing works
“It uses Matterport3D environment scans (Chang et al. 2017), and creates a realistic simulation of densely sampled grid of the 3D space with 1 m resolution driven by the Habitat-Simulator engine (Savva et al. 2019).”
MethodAnderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sunderhauf, N., Reid, I., Gould, S., & van den Hengel, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00387 · Cited by 1,187 · see citing works
Li, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908 · see citing works
Wang, X., Huang, Q., Celikyilmaz, A., Gao, J., Shen, D., Wang, Y.-F., Wang, W. Y., & Zhang, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00679 · Cited by 522 · cited 1× · see citing works
“To tackle the task, Wang et al. (Wang et al. 2019) uses cross-modal attention to focus on the relevant parts of both vision and language modalities, while others (Fried et al. 2018; Tan et al. 2019), used augmented instruction-trajectory pairs to improve the VLN performance.”
Matterport3D: Learning from RGB-D Data in Indoor EnvironmentsPreprint
Chang, A., Dai, A., Funkhouser, T. A., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.06158 · Cited by 332 · see citing works
Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout
Tan, H., Yu, L., & Bansal, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1268 · Cited by 293 · cited 1× · see citing works
“To tackle the task, Wang et al. (Wang et al. 2019) uses cross-modal attention to focus on the relevant parts of both vision and language modalities, while others (Fried et al. 2018; Tan et al. 2019), used augmented instruction-trajectory pairs to improve the VLN performance.”
Speaker-Follower Models for Vision-and-Language Navigation
Fried, D., Hu, R., Cirik, V., Rohrbach, A., Andreas, J., Morency, L.-P., Berg-Kirkpatrick, T., Saenko, K., Klein, D., & Darrell, T. (2018) · arXiv · doi:10.48550/arxiv.1806.02724 · Cited by 291 · cited 2× · see citing works
“To tackle the task, Wang et al. (Wang et al. 2019) uses cross-modal attention to focus on the relevant parts of both vision and language modalities, while others (Fried et al. 2018; Tan et al. 2019), used augmented instruction-trajectory pairs to improve the VLN performance.”
BackgroundVLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation
Hong, Y., Wu, Q., Qi, Y., Rodriguez-Opazo, C., & Gould, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00169 · Cited by 264 · cited 5× · see citing works
“The task in VLN is to use (or execute) natural language instructions to reach a target location.”
BackgroundMethodInfluentialConversational AI: The Science Behind the Alexa PrizePreprint
Ram, A., Prasad, R., Khatri, C., Venkatesh, A., Gabriel, R., Liu, Q., Nunn, J., Hedayatnia, B., Cheng, M., Nagar, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.03604 · Cited by 261 · see citing works
DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion FramesPreprint
Wijmans, E., Kadian, A., Morcos, A. S., Lee, S., Essa, I. A., Parikh, D., Savva, M., & Batra, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00357 · Cited by 253 · cited 1× · see citing works
“Using this reward setup, the policies are trained with the DD-PPO algo-rithm (Wijmans et al. 2019).”
MethodSiddhant, A., & Lipton, Z. C. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1318 · Cited by 239 · cited 1× · see citing works
“Note that, some of these challenges are partially addressed in prior works (Kesiraju et al. 2020; Siddhant and Lipton 2018; Xiao and Wang 2019) such as (v) and (vi).”
BackgroundVisual Programming: Compositional visual reasoning without training
Gupta, T., & Kembhavi, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01436 · Cited by 230 · cited 1× · see citing works
“…deep neural networks and sophisticated language models have led to significant advancements in building conversational agents that can collaborate with humans in solving challenging reasoning tasks (Peng et al. 2023; Ram et al. 2018; Chowdhery et al. 2022; Gupta and Kembhavi 2023; You et al. 2022).”
BackgroundSoundSpaces: Audio-Visual Navigation in 3D Environments
Chen, C., Jain, U., Schissler, C., Gari, S. V. A., Al-Halah, Z., Ithapu, V. K., Robinson, P., & Grauman, K. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_2 · Cited by 202 · cited 10× · see citing works
“Task Setup: We assume the standard embodied audio goal problem setup (Chen et al. 2020), where the agent is equipped with an RGBD camera and a binaural microphone and at any time step can take one of four navigation actions: { stop , move forward , turn right , turn left } in a densely-sampled 3D…”
BackgroundMethodInfluentialImproving Vision-and-Language Navigation with Image-Text Pairs from the Web
Majumdar, A., Shrivastava, A., Lee, S., Anderson, P., Parikh, D., & Batra, D. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_16 · Cited by 201 · cited 2× · see citing works
“Recent approaches have begun using transformer-based architectures, such as BERT (Devlin et al. 2018) for VLN (Hong et al. 2021; Majumdar et al. 2020).”
BackgroundMethodSelf-Monitoring Navigation Agent via Auxiliary Progress Estimation
Ma, C.-Y., Lu, J., Wu, Z., AlRegib, G., Kira, Z., Socher, R., & Xiong, C. (2019) · arXiv · doi:10.48550/arxiv.1901.03035 · Cited by 200 · cited 1× · see citing works
“Instruction-at-start is a well-explored research area (Anderson et al. 2018; Hong et al. 2021; Ke et al. 2019; Liu et al. 2021; Majumdar et al. 2020; Ma et al. 2019a,b; Zhu et al. 2020; Chen et al. 2021c; Pashevich et al. 2021; Guhur et al. 2021) in which the agent is given a language instruction…”
BackgroundPeng, B., Li, C., He, P., Galley, M., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2304.03277 · Cited by 183 · see citing works
LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action
Shah, D., Osinski, B., Ichter, B., & Levine, S. (2022) · arXiv · doi:10.48550/arxiv.2207.04429 · Cited by 179 · cited 1× · see citing works
“In the context of Vision and Language Navigation, early works like LM-Nav (Shah et al. 2022) analyzed landmarks in the instruction to be used for visual navigation.”
MethodScene Memory Transformer for Embodied Agents in Long-Horizon Tasks
Fang, K., Toshev, A., Fei-Fei, L., & Savarese, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00063 · Cited by 178 · see citing works
Thomason, J., Murray, M., Cakmak, M., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1907.04957 · Cited by 173 · cited 1× · see citing works
“Thomason et al. (Thomason et al. 2020) introduced the CVDN dataset with human-human dialogue for navigation.”
BackgroundThe Regretful Agent: Heuristic-Aided Navigation Through Progress Estimation
Ma, C., Wu, Z., AlRegib, G., Xiong, C., & Kira, Z. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00689 · Cited by 166 · cited 1× · see citing works
“Instruction-at-start is a well-explored research area (Anderson et al. 2018; Hong et al. 2021; Ke et al. 2019; Liu et al. 2021; Majumdar et al. 2020; Ma et al. 2019a,b; Zhu et al. 2020; Chen et al. 2021c; Pashevich et al. 2021; Guhur et al. 2021) in which the agent is given a language instruction…”
BackgroundTactical Rewind: Self-Correction via Backtracking in Vision-And-Language Navigation
Ke, L., Li, X., Bisk, Y., Holtzman, A., Gan, Z., Liu, J., Gao, J., Choi, Y., & Srinivasa, S. S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00690 · Cited by 148 · cited 1× · see citing works
“Instruction-at-start is a well-explored research area (Anderson et al. 2018; Hong et al. 2021; Ke et al. 2019; Liu et al. 2021; Majumdar et al. 2020; Ma et al. 2019a,b; Zhu et al. 2020; Chen et al. 2021c; Pashevich et al. 2021; Guhur et al. 2021) in which the agent is given a language instruction…”
BackgroundAirbert: In-domain Pretraining for Vision-and-Language Navigation
Guhur, P.-L., Tapaswi, M., Chen, S., Laptev, I., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00166 · Cited by 139 · cited 1× · see citing works
“…area (Anderson et al. 2018; Hong et al. 2021; Ke et al. 2019; Liu et al. 2021; Majumdar et al. 2020; Ma et al. 2019a,b; Zhu et al. 2020; Chen et al. 2021c; Pashevich et al. 2021; Guhur et al. 2021) in which the agent is given a language instruction at its start describing the intended path.”
BackgroundQuantifying Uncertainties in Natural Language Processing Tasks
Xiao, Y., & Wang, W. Y. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017322 · Cited by 136 · cited 1× · see citing works
“Note that, some of these challenges are partially addressed in prior works (Kesiraju et al. 2020; Siddhant and Lipton 2018; Xiao and Wang 2019) such as (v) and (vi).”
BackgroundEpisodic Transformer for Vision-and-Language Navigation
Pashevich, A., Schmid, C., & Sun, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01564 · Cited by 135 · cited 1× · see citing works
“…area (Anderson et al. 2018; Hong et al. 2021; Ke et al. 2019; Liu et al. 2021; Majumdar et al. 2020; Ma et al. 2019a,b; Zhu et al. 2020; Chen et al. 2021c; Pashevich et al. 2021; Guhur et al. 2021) in which the agent is given a language instruction at its start describing the intended path.”
BackgroundHistory Aware Multimodal Transformer for Vision-and-Language NavigationPreprint
Chen, S., Guhur, P.-L., Schmid, C., & Laptev, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.13309 · Cited by 124 · cited 1× · see citing works
“…research area (Anderson et al. 2018; Hong et al. 2021; Ke et al. 2019; Liu et al. 2021; Majumdar et al. 2020; Ma et al. 2019a,b; Zhu et al. 2020; Chen et al. 2021c; Pashevich et al. 2021; Guhur et al. 2021) in which the agent is given a language instruction at its start describing the intended…”
BackgroundLearning to Answer Questions in Dynamic Audio-Visual Scenarios
Li, G., Wei, Y., Tian, Y., Xu, C., Wen, J.-R., & Hu, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01852 · Cited by 124 · cited 1× · see citing works
“While, much effort has been expended on tasks that are predominantly in the language domain, such progress is yet to percolate into real world problems that need complex reasoning over multiple modalities of perception (Li et al. 2022; Liu et al. 2023).”
BackgroundLook, Listen, and Act: Towards Audio-Visual Embodied Navigation
Gan, C., Zhang, Y., Wu, J., Gong, B., & Tenenbaum, J. B. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197008 · Cited by 124 · cited 2× · see citing works
“Audio-Visual Embodied Navigation Tasks: Recent years have seen several works in Embodied AI that consider the audio-goal navigation task (Chen et al. 2020, 2021a; Gan et al. 2020; Yu et al. 2022).”
BackgroundMethodNguyen, K., & Daumé III, H. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1063 · Cited by 115 · cited 1× · see citing works
“In the oracle guidance setting, an agent may receive feedback (ground truth actions (Chi et al. 2020), encoded ground truth action (Nguyen et al. 2019a), or a fixed set of natural language instructions (Nguyen et al. 2019b)) from an oracle during navigation.”
BackgroundNguyen, K., Dey, D., Brockett, C., & Dolan, B. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01281 · Cited by 101 · cited 1× · see citing works
“In the oracle guidance setting, an agent may receive feedback (ground truth actions (Chi et al. 2020), encoded ground truth action (Nguyen et al. 2019a), or a fixed set of natural language instructions (Nguyen et al. 2019b)) from an oracle during navigation.”
BackgroundSemantic Audio-Visual Navigation
Chen, C., Al-Halah, Z., & Grauman, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01526 · Cited by 96 · see citing works
Vision-Language Navigation with Random Environmental Mixup
Liu, C., Zhu, F., Chang, X., Liang, X., Ge, Z., & Shen, Y.-D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00167 · Cited by 80 · cited 2× · see citing works
“Instruction-at-start is a well-explored research area (Anderson et al. 2018; Hong et al. 2021; Ke et al. 2019; Liu et al. 2021; Majumdar et al. 2020; Ma et al. 2019a,b; Zhu et al. 2020; Chen et al. 2021c; Pashevich et al. 2021; Guhur et al. 2021) in which the agent is given a language instruction…”
BackgroundEnd-to-end Spoken Conversational Question Answering: Task, Dataset and Model
You, C., Chen, N., Liu, F., et al. (2022) · NAACL-HLT · doi:10.48550/arxiv.2204.14272 · Cited by 63 · cited 1×
“…deep neural networks and sophisticated language models have led to significant advancements in building conversational agents that can collaborate with humans in solving challenging reasoning tasks (Peng et al. 2023; Ram et al. 2018; Chowdhery et al. 2022; Gupta and Kembhavi 2023; You et al. 2022).”
BackgroundBabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps
Zhu, W., Hu, H., Chen, J., Deng, Z., Jain, V., Ie, E., & Sha, F. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.229 · Cited by 60 · cited 1× · see citing works
“…is a well-explored research area (Anderson et al. 2018; Hong et al. 2021; Ke et al. 2019; Liu et al. 2021; Majumdar et al. 2020; Ma et al. 2019a,b; Zhu et al. 2020; Chen et al. 2021c; Pashevich et al. 2021; Guhur et al. 2021) in which the agent is given a language instruction at its start…”
BackgroundLe, T. P., Vien, N. A., & Chung, T. (2018) · IEEE Access · doi:10.1109/access.2018.2854283 · Cited by 59 · cited 1× · see citing works
“As we envisage CAVEN to incorporate various modules with diverse temporal spans, it is natural to consider a partially observable semi-Markov decision process (POSMDP) as our control module (Le et al. 2018).”
Sound Adversarial Audio-Visual NavigationPreprint
Yu, Y., Huang, W.-B., Sun, F.-C., Chen, C., Wang, Y.-K., & Liu, X. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2202.10910 · Cited by 58 · see citing works
Self-citationJust Ask: An Interactive Learning Framework for Vision and Language Navigation
Chi, T.-C., Shen, M., Eric, M., Kim, S., & Hakkani-Tur, D. Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i03.5627 · Cited by 53 · cited 2× · see citing works
“In the oracle guidance setting, an agent may receive feedback (ground truth actions (Chi et al. 2020), encoded ground truth action (Nguyen et al. 2019a), or a fixed set of natural language instructions (Nguyen et al. 2019b)) from an oracle during navigation.”
BackgroundMethodLearning to Set Waypoints for Audio-Visual Navigation
Chen, C., Majumder, S., Al-Halah, Z., Gao, R., Ramakrishnan, S. K., & Grauman, K. (2020) · arXiv · doi:10.48550/arxiv.2008.09622 · Cited by 53 · see citing works
Multimodal Transformer with Variable-Length Memory for Vision-and-Language Navigation
Lin, C., Jiang, Y., Cai, J., Qu, L., Haffari, G., & Yuan, Z. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_22 · Cited by 35 · cited 1× · see citing works
“In the bi-directional interaction setting, an agent can use natural language to seek navigation help (Banerjee et al. 2021; Thoma-son et al. 2020; Cao et al. 2022; Lin et al. 2022).”
BackgroundEnd-to-end Spoken Conversational Question Answering: Task, Dataset and Model
You, C., Chen, N., Liu, F., Ge, S., Wu, X., & Zou, Y. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.91 · Cited by 26 · see citing works
Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering
Liu, X., Dong, Z., & Peng, Z. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00442 · Cited by 21 · cited 1× · see citing works
“While, much effort has been expended on tasks that are predominantly in the language domain, such progress is yet to percolate into real world problems that need complex reasoning over multiple modalities of perception (Li et al. 2022; Liu et al. 2023).”
BackgroundSelf-citationSelf-Motivated Communication Agent for Real-World Vision-Dialog Navigation
Zhu, Y., Weng, Y., Zhu, F., Liang, X., Ye, Q., Lu, Y., & Jiao, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00162 · Cited by 21 · cited 1× · see citing works
“Self-Motivated Communication Agent (SCoA) (Zhu et al. 2021) permits the agent to only ask templated questions filled in with labels of detected scene objects, grossly limiting the nature of interaction be-tween the agent and the oracle.”
BackgroundAVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments
Paul, S., Roy-Chowdhury, A. K., & Cherian, A. (2022) · arXiv · doi:10.48550/arxiv.2210.07940 · Cited by 13 · cited 8× · see citing works
“…attention in the recent years (Chen et al., 2021), we consider a variant of this task, dubbed audio-visual-language embodied navigation (AVLEN) (Paul et al., 2022), where the agent has the ability to interact with a human/oracle when it is unable to solve the task by itself and potentially a r…”
BackgroundMethodInfluentialSelf-citationThe RobotSlang Benchmark: Dialog-guided Robot Localization and NavigationPreprint
Banerjee, S., Thomason, J., & Corso, J. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12639 · Cited by 11 · see citing works
Learning Document Embeddings Along With Their Uncertainties
Kesiraju, S., Plchot, O., Burget, L., & Gangashetty, S. V. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3012062 · Cited by 11 · cited 1× · see citing works
“Note that, some of these challenges are partially addressed in prior works (Kesiraju et al. 2020; Siddhant and Lipton 2018; Xiao and Wang 2019) such as (v) and (vi).”
BackgroundNavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language ModelsPreprint
Zhou, G., Hong, Y., & Wu, Q. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.16986 · Cited by 11 · see citing works
BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby StepsPreprint
Zhu, W., Hu, H., Chen, J., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.04625 · Cited by 8 · see citing works
Learning to Answer Questions in Dynamic Audio-Visual ScenariosPreprint
Li, G., Wei, Y., Tian, Y., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.14072 · Cited by 5 · see citing works
Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-AnsweringPreprint
Liu, X., Dong, Z., & Zhang, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.06238 · Cited by 4 · see citing works
Self-citationLearning to Navigate Unseen Environments: Back Translation with Environmental DropoutPreprint
Tan, H. X., Yu, L., & Bansal, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04195
Goal-oriented Vision-and-Dialog Navigation via Reinforcement Learning
Cao, Y. L., Lu, K., DeFazio, D., & Zhang, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.327
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
Gu, J., Stefani, E., Wu, Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.12667
Siddhant, A., & Lipton, Z. C. (2018) · arXiv · doi:10.48550/arxiv.1808.05697
Flagged first, then by citations
Research works printed but not linked: 2
- [3]3. PMLR.
- [20]20. He, K.; et al. 2021. Landmark-RxR: Solving Vision-and- Language Navigation with Fine-Grained Alignment Super- vision. In Ranzato, M.; and etal, eds., Advances in Neural Information Processing Systems, volume 34, 652-663.
Cited by
Sorted by citations
Hearing Anywhere in Any Environment
Liu, X., Kumar, A. K., Calamia, P. T., Garí, S. V. A., Murdock, C., Ananthabhotla, I., Robinson, P. W., Shlizerman, E., Ithapu, V. K., & Gao, R. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00538 · Cited by 7 · see citing works
“…spatial cues from the environment [16, 19, 20, 26, 34], audio-visual navigation in environments with varying room acoustic properties [3, 4, 15, 18, 29], learning image features, scene structures, or human locations from echoes [12, 17], ambient sound [9], or music [54] in the room, and using RGB…”
Self-citationTaioli, F., Zorzi, E., Franchi, G., Castellini, A., Farinelli, A., Cristani, M., & Wang, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01745 · Cited by 4 · see citing works
“Common approaches involve agents asking users for help [55], with responses typically consisting of shortest-path actions to target [7, 51] or simple natural language sub-goals guiding navigation [30, 35, 36, 40, 45].”
Embodied Navigation with Auxiliary Task of Action Description Prediction
Kondoh, H., & Kanezaki, A. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00660
“[44] introduced language into audio-visual navigation as instructions to reach the goal.”
- Authors
- Liu, X., Paul, S., Chatterjee, M., & Cherian, A.
- Affiliations
- University of Washington (US)
- Mitsubishi Electric Research Laboratories (United States) (US)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2024-03-24
- Volume
- 38
- Issue
- 4
- Pages
- 3765-3773
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W4393149068
- Cited by
- 6
- References
- 56
- Citations by year
- 2025: 5 · 2024: 1
- FWCI
- 1.73
- Type
- conference paper
- Language
- en
- Open access
- diamond
- dx.doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Engineering › Automotive Engineering › Spatial Cognition and Navigation
- Keywords
- embodied conversational agents, noisy environments, semi-Markov decision process, trajectory prediction, question generation, natural language reasoning, sound localization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-01.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-01.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
