Can an Embodied Agent Find Your “Cat-shaped Mug”? LLM-Based Zero-Shot Object Navigation
Open accessThe DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
We present language-guided exploration (LGX), a novel algorithm forLanguage-Driven Zero-Shot Object Goal Navigation(L-ZSON), where an embodied agent navigates to anuniquely describedtarget object in apreviously unseenenvironment. Our approach makes use of large language models (LLMs) for this task by leveraging the LLM's commonsense-reasoning capabilities for making sequential navigational decisions. Simultaneously, we perform generalized target object detection using a pre-trained Vision-Language grounding model. We achieve state-of-the-art zero-shot object navigation results on RoboTHOR with a success rate (SR) improvement of over 27% over the current baseline of the OWL-ViT CLIP on Wheels (OWL CoW). Furthermore, we study the usage of LLMs for robot navigation and present an analysis of various prompting strategies affecting the model output. Finally, we showcase the benefits of our approach viareal-worldexperiments that indicate the superior performance of LGX in detecting and navigating to visually unique objects.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.00982 · Cited by 3,653
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Li, J., Li, D., Xiong, C., & Hoi, S. (2022) · arXiv · doi:10.48550/arxiv.2201.12086 · Cited by 1,741
A frontier-based approach for autonomous exploration
Yamauchi, B. (2002) · Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA'97. 'Towards New Computational Principles for Robotics and Automation' · doi:10.1109/cira.1997.613851 · Cited by 1,695
Habitat: A Platform for Embodied AI Research
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00943 · Cited by 1,237 · cited 1×
“Those designed for object navigation tasks, including RoboTHOR [16] and AI Habitat [17], only contain common day-to-day household objects described using simple language (eg.”
BackgroundDo As I Can, Not As I Say: Grounding Language in Robotic Affordances
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.01691 · Cited by 624
Cited by
Agentic Large Language Models, a Survey
Plaat, A., Van Duijn, M., Van Stein, N., Preuss, M., Van der Putten, P., & Batenburg, K. J. (2025) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.18675 · Cited by 75
Bridging Zero-shot Object Navigation and Foundation Models through Pixel-Guided Navigation Skill
Cai, W., Huang, S., Cheng, G., Long, Y., Gao, P., Sun, C., & Dong, H. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10610499 · Cited by 63
“But for the locomotion part, most of the zero-shot object navigation works still tethered to the map-based planning approaches [7], [8], [9], [10], [11].”
BackgroundMethodA knowledge-driven framework for Robotic Odor Source Localization using large language models
Mahmud, K. R., Wang, L., Hassan, S., & Zhang, Z. (2025) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2025.104915 · Cited by 10
SignNav: Directional Signage-Cue Guided Visual Navigation in Large-Scale Indoor Environments
Sun, J., Huang, Y., Li, H., Xiao, S., Guo, S., Ghaffari, M., Li, Q., Xu, C., & Kong, H. (2026) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2026.3730302
Metadata
- Authors
- Dorbala, V. S., Mullen, J. F., & Manocha, D.
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2024-05
- Volume
- 9
- Issue
- 5
- Pages
- 4083-4090
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W4390204337
- Cited by
- 83
- References
- 22
- FWCI
- 9.02
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Can an Embodied Agent Find Your “Cat-shaped Mug”? LLM-Based Zero-Shot Object Navigation. https://app.cochat.ai/papers/10.1109/lra.2023.3346800
- The paper
- Dorbala, V. S., Mullen, J. F., & Manocha, D. (2024). Can an Embodied Agent Find Your “Cat-shaped Mug”? LLM-Based Zero-Shot Object Navigation. IEEE Robotics and Automation Letters. https://doi.org/10.1109/lra.2023.3346800
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.