The DOI matches its Crossref record; 37 of the 41 references it cites were found in a registry or an index.
Bridging Zero-shot Object Navigation and Foundation Models through Pixel-Guided Navigation Skill
Open accessAbstract
Zero-shot object navigation is a challenging task for home-assistance robots. This task emphasizes visual grounding, commonsense inference and locomotion abilities, where the first two are inherent in foundation models. But for the locomotion part, most works still depend on map-based planning approaches. The gap between RGB space and map space makes it difficult to directly transfer the knowledge from foundation models to navigation tasks. In this work, we propose a Pixel-guided Navigation skill (PixNav), which bridges the gap between the foundation models and the embodied navigation task. It is straightforward for recent foundation models to indicate an object by pixels, and with pixels as the goal specification, our method becomes a versatile navigation policy towards all different kinds of objects. Besides, our PixNav is a pure RGB-based policy that can reduce the cost of homeassistance robots. Experiments demonstrate the robustness of the PixNav which achieves 80+% success rate in the local path-planning task. To perform long-horizon object navigation, we design an LLM-based planner to utilize the commonsense knowledge between objects and rooms to select the best waypoint. Evaluations across both photorealistic indoor simulators and real-world environments validate the effectiveness of our proposed navigation strategy. More details are accessible via our project website https://sites.google.com/view/pixnav/.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Despite their strong performance in simulated environments, these methods are highly susceptible to data distribution shifts between training and testing environments, severely impairing their generalization to real-world physical deployments.”
From Brain-inspired spatial intelligence for embodied agents (2026) · cites this paper as [11]
In the citing paper
“While they streamline the decision-making process, they must simultaneously learn spatial memory tracking and action planning, necessitating extensive trajectory data or expert demonstrations for effective training.”
From Brain-inspired spatial intelligence for embodied agents (2026) · cites this paper as [11]
In the citing paper
“For instance, PixNav 11 performs greedy navigation by predicting optimal actions toward salient pixels in the current view.”
From Brain-inspired spatial intelligence for embodied agents (2026) · cites this paper as [11]
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“As for language, LLM shows surprising performance on almost all the tasks, including text understanding, generation [26], [27] and even planning for robotics field [28], [29], [30], [31].”
BackgroundLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 2×
“ZSON [20] transfers an image-goal navigation policy for object-goal navigation tasks with the text-image alignment from the CLIP [21] encoder.”
BackgroundMethodKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953 · cited 3×
“SAM [3] pioneers in delivering fine-granular segmentation for a diverse set of objects.”
BackgroundMethodInternational Conference on Learning Representations (ICLR 2013)
松尾, 豊. (2013) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 6,286
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1×
“As for language, LLM shows surprising performance on almost all the tasks, including text understanding, generation [26], [27] and even planning for robotics field [28], [29], [30], [31].”
BackgroundInternational Conference on Machine Learning (ICML)-2005
津田, 宏. (2006) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 2,558
Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72970-6_3 · Cited by 2,074
Habitat: A Platform for Embodied AI Research
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00943 · Cited by 1,237 · cited 1×
“In our work, we use the Habitat-Lab [36], [37] with HM3D [38] dataset to generate the training trajectories for PixNav.”
Method(2022) · doi:10.15607/rss.2023.xix.025 · Cited by 916
Research works printed but not linked: 6
- Zhang, Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention
- Walke, Bridgedata v2: A dataset for robot learning at scale
- Rana, Sayplan: Grounding large language models using 3d scene graphs for scalable task planning
- Zhu, Minigpt-4: Enhancing vision-language understanding with advanced large language models
- Driess, Palme: An embodied multimodal language model
- 2023, Gpt-4 technical report, ArXiv, abs/2303.08774 · arXiv
Cited by
A Survey of Object Goal Navigation
Sun, J., Wu, J., Ji, Z., & Lai, Y. (2024) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2024.3378010 · Cited by 82
“Some works [73], [74], [75], [76], [77] also apply Large Language Models (LLMs) in such methods.”
Affordances-Oriented Planning Using Foundation Models for Continuous Vision-Language Navigation
Chen, J., Lin, B., Liu, X., Ma, L., Liang, X., & Wong, K.-Y. K. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i22.34526 · Cited by 117
Huang, S., Ponomarenko, I., Jiang, Z., Li, X., Hu, X., Gao, P., Li, H., & Dong, H. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801993 · Cited by 64
“Another research direction leverages the reasoning capabilities of large language models (LLMs) for planning [18], [19] or even directly generating the policy codes [3], [20], [21] by prompt engineering.”
Self-citationTriHelper: Zero-Shot Object Navigation with Dynamic Assistance
Zhang, L., Zhang, Q., Wang, H., Xiao, E., Jiang, Z., Chen, H., & Xu, R. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802670 · Cited by 72
“The PixNav[13] project, through the use of foundational models and specifying navigation targets in pixel units, achieves universal navigation for various object types.”
UniGoal: Towards Universal Zero-shot Goal-oriented Navigation
Yin, H. H., Xu, X., Zhao, L., Wang, Z., Zhou, J., & Lu, J. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01775 · Cited by 130
“According to the goal type, zero-shot navigation can be mainly divided into ON [3, 46], IIN [16] and TN [25].”
Metadata
- Authors
- Cai, W., Huang, S., Cheng, G., Long, Y., Gao, P., Sun, C., & Dong, H.
- Affiliations
- Southeast University (BD)
- Shanghai Artificial Intelligence Laboratory (CN)
- Peking University (CN)
- Venue
- 2024 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2024-05-13
- Pages
- 5228-5234
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W4401417236
- Cited by
- 63
- References
- 41
- Citations by year
- 2026: 27 · 2025: 31 · 2024: 5
- FWCI
- 18.59
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotics and Sensor-Based Localization
- Keywords
- foundation models, commonsense reasoning, visual grounding, local path planning, waypoint selection, embodied AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Bridging Zero-shot Object Navigation and Foundation Models through Pixel-Guided Navigation Skill. https://app.cochat.ai/papers/10.1109/icra57147.2024.10610499
- The paper
- Cai, W., Huang, S., Cheng, G., Long, Y., Gao, P., Sun, C., & Dong, H. (2024). Bridging Zero-shot Object Navigation and Foundation Models through Pixel-Guided Navigation Skill. 2024 IEEE International Conference on Robotics and Automation (ICRA). https://doi.org/10.1109/icra57147.2024.10610499
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.