Can an Embodied Agent Find Your “Cat-shaped Mug”? LLM-Based Zero-Shot Object Navigation
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
We present language-guided exploration (LGX), a novel algorithm forLanguage-Driven Zero-Shot Object Goal Navigation(L-ZSON), where an embodied agent navigates to anuniquely describedtarget object in apreviously unseenenvironment. Our approach makes use of large language models (LLMs) for this task by leveraging the LLM's commonsense-reasoning capabilities for making sequential navigational decisions. Simultaneously, we perform generalized target object detection using a pre-trained Vision-Language grounding model. We achieve state-of-the-art zero-shot object navigation results on RoboTHOR with a success rate (SR) improvement of over 27% over the current baseline of the OWL-ViT CLIP on Wheels (OWL CoW). Furthermore, we study the usage of LLMs for robot navigation and present an analysis of various prompting strategies affecting the model output. Finally, we showcase the benefits of our approach viareal-worldexperiments that indicate the superior performance of LGX in detecting and navigating to visually unique objects.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.00982 · Cited by 3,653
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · cited 1×
“Thedownstreamtransferofsuch‘foundationmodels’[13]has shown great improvement in various vision and language tasks such as image captioning [14], question answering [15], and the open-vocabulary target grounding [10] required in L-ZSON.”
BackgroundLi, J., Li, D., Xiong, C., & Hoi, S. (2022) · arXiv · doi:10.48550/arxiv.2201.12086 · Cited by 1,741
A frontier-based approach for autonomous exploration
Yamauchi, B. (2002) · Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA'97. 'Towards New Computational Principles for Robotics and Automation' · doi:10.1109/cira.1997.613851 · Cited by 1,695
Habitat: A Platform for Embodied AI Research
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00943 · Cited by 1,237 · cited 1×
“Those designed for object navigation tasks, including RoboTHOR [16] and AI Habitat [17], only contain common day-to-day household objects described using simple language (eg.”
BackgroundDo As I Can, Not As I Say: Grounding Language in Robotic Affordances
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.01691 · Cited by 624
On evaluation of embodied navigation agents
Anderson, P., Chang, A., Chaplot, D. S., Dosovitskiy, A., Gupta, S., Koltun, V., Kosecka, J., Malik, J., Mottaghi, R., Savva, M., et al. (2018) · arXiv · doi:10.48550/arxiv.1807.06757 · Cited by 622
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models
Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., & Garg, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10161317 · Cited by 613 · cited 1×
“The latter uses GPT-3 [37] to obtain “landmarks” or subgoals, while the former focuses on using an LLM for “code-writing” [40].”
MethodLarge Language Models Are Human-Level Prompt Engineers
Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2022) · arXiv · doi:10.48550/arxiv.2211.01910 · Cited by 330 · cited 1×
“The performance of LLMs is highly dependent on the quality of the prompts used [21] as input.”
BackgroundObject Goal Navigation using Goal-Oriented Semantic Exploration
Chaplot, D. S., Gandhi, D., Gupta, A., & Salakhutdinov, R. (2020) · arXiv · doi:10.48550/arxiv.2007.00643 · Cited by 321 · cited 1×
“Some recent works address this through the ZSON task [7], [8], [9], which enables the sequential decision making component to generalize to new locations, in part by removing environment-specific training.”
BackgroundLanguage Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
Huang, W., Abbeel, P., Pathak, D., & Mordatch, I. (2022) · arXiv · doi:10.48550/arxiv.2201.07207 · Cited by 304 · cited 1×
“Several recent works [31], [32], [33] have also used LLMs specifically for their planning capabilities.”
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform
Deitke, M., Han, W., Herrasti, A., Kembhavi, A., Kolve, E., Mottaghi, R., Salvador, J., Schwenk, D., VanderBilt, E., Wallingford, M., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00323 · Cited by 220 · cited 2×
“Those designed for object navigation tasks, including RoboTHOR [16] and AI Habitat [17], only contain common day-to-day household objects described using simple language (eg.”
BackgroundMethodLM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action
Shah, D., Osinski, B., Ichter, B., & Levine, S. (2022) · arXiv · doi:10.48550/arxiv.2207.04429 · Cited by 179 · cited 1×
“A few recent works have looked at using generative models for navigation, specifically, LM-Nav [38] and VLMaps [39].”
BackgroundObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects
Batra, D., Gokaslan, A., Kembhavi, A., Maksymets, O., Mottaghi, R., Savva, M., Toshev, A., & Wijmans, E. (2020) · arXiv · doi:10.48550/arxiv.2006.13171 · Cited by 177 · cited 1×
“State-of-the-art approaches to solving Object Goal Navigation are based on fully supervised learning [4], [5], [6], which is not practical for an agent that is expected to detect arbitrarily described objects and perform consistently in dynamic real-world environments.”
BackgroundThomason, J., Murray, M., Cakmak, M., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1907.04957 · Cited by 173 · cited 1×
“[26],[27]andGaoetal.”
PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning
Ramakrishnan, S. K., Chaplot, D. S., Al-Halah, Z., Malik, J., & Grauman, K. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01832 · Cited by 154 · cited 6×
“In Embodied AI, using language to provide navigational guidance towards a target object presents many popular tasks including Vision-and-Language Navigation (VLN) [32], [33] and Object Goal Navigation [4], [5].”
BackgroundMethodInfluentialLarge language models as commonsense knowledge for large-scale task planning
Zhao, Z., Lee, W. S., & Hsu, D. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1387 · Cited by 95
Dialfred: Dialogue-enabled agents for embodied instruction following
Gao, X., Gao, Q., Gong, R., Lin, K., Thattai, G., & Sukhatme, G. S. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3193254 · Cited by 55
(2023) · doi:10.1109/iros55552.2023.10341959 · Cited by 4
Cited by
Agentic Large Language Models, a Survey
Plaat, A., Van Duijn, M., Van Stein, N., Preuss, M., Van der Putten, P., & Batenburg, K. J. (2025) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.18675 · Cited by 75
Bridging Zero-shot Object Navigation and Foundation Models through Pixel-Guided Navigation Skill
Cai, W., Huang, S., Cheng, G., Long, Y., Gao, P., Sun, C., & Dong, H. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10610499 · Cited by 63
“But for the locomotion part, most of the zero-shot object navigation works still tethered to the map-based planning approaches [7], [8], [9], [10], [11].”
BackgroundMethodA knowledge-driven framework for Robotic Odor Source Localization using large language models
Mahmud, K. R., Wang, L., Hassan, S., & Zhang, Z. (2025) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2025.104915 · Cited by 10
SignNav: Directional Signage-Cue Guided Visual Navigation in Large-Scale Indoor Environments
Sun, J., Huang, Y., Li, H., Xiao, S., Guo, S., Ghaffari, M., Li, Q., Xu, C., & Kong, H. (2026) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2026.3730302
Metadata
- Authors
- Dorbala, V. S., Mullen, J. F., & Manocha, D.
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2024-05
- Volume
- 9
- Issue
- 5
- Pages
- 4083-4090
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W4390204337
- Cited by
- 83
- References
- 23
- FWCI
- 9.02
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.