Navigating to objects in the real world
Open accessThe DOI matches its Crossref record; 106 of the 106 references it cites were found in a registry or an index.
Abstract
Semantic navigation is necessary to deploy mobile robots in uncontrolled environments such as homes or hospitals. Many learning-based approaches have been proposed in response to the lack of semantic understanding of the classical pipeline for spatial navigation, which builds a geometric map using depth sensors and plans to reach point goals. Broadly, end-to-end learning approaches reactively map sensor inputs to actions with deep neural networks, whereas modular learning approaches enrich the classical pipeline with learning-based semantic sensing and exploration. However, learned visual navigation policies have predominantly been evaluated in sim, with little known about what works on a robot. We present a large-scale empirical study of semantic visual navigation methods comparing representative methods with classical, modular, and end-to-end learning approaches across six homes with no prior experience, maps, or instrumentation. We found that modular learning works well in the real world, attaining a 90% success rate. In contrast, end-to-end learning does not, dropping from 77% sim to a 23% real-world success rate because of a large image domain gap between sim and reality. For practitioners, we show that modular learning is a reliable approach to navigate to objects: Modularity and abstraction in policy design enable sim-to-real transfer. For researchers, we identify two key issues that prevent today’s simulators from being reliable evaluation benchmarks—a large sim-to-real gap in images and a disconnect between sim and real-world error modes—and propose concrete steps forward.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Low-level navigation policies are typically derived from deterministic path planning conditioned on environmental constraints and high-level spatial goals52.”
From Brain-inspired spatial intelligence for embodied agents (2026) · cites this paper as [52]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 3×
“Depth features were computed with a ResNet50 pretrained on point goal navigation (44).”
MethodHuman-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 1×
“Inspired by seminal proofs of concept in autonomous driving [26, 27] and early successes of pixel-to-action deep reinforcement learning [28, 29], early applications of end-to-end learning to semantic navigation include [30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40].”
BackgroundHe, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · cited 4×
“RGB, depth, and semantic frames are then fed into a GRU [109] along with other low dimensional features — the robot pose, a learnable goal object embedding, the fraction of the visual input occupied by the goal category, and the previous action — to predict a distribution over next actions.”
BackgroundInfluentialContinuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · cited 1×
“Inspired by seminal proofs of concept in autonomous driving [26, 27] and early successes of pixel-to-action deep reinforcement learning [28, 29], early applications of end-to-end learning to semantic navigation include [30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40].”
BackgroundA robust layered control system for a mobile robot
Brooks, R. (1986) · IEEE Journal on Robotics and Automation · doi:10.1109/jra.1986.1087032 · Cited by 7,799
KinectFusion: Real-time dense surface mapping and tracking
Newcombe, R. A., Fitzgibbon, A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., & Hodges, S. (2011) · 2011 10th IEEE International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2011.6092378 · Cited by 4,008 · cited 1×
“Among approaches that can operate in unseen environments, classical approaches typically build a geometric map of the environment using depth sensors [12, 13, 14], and later a monocular RGB camera [15, 16, 17], while simultaneously localizing the robot relative to its growing map.”
BackgroundMonoSLAM: Real-time single camera SLAM
Davison, A. J., Reid, I. D., Molton, N. D., & Stasse, O. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1049 · Cited by 3,959 · cited 1×
“Among approaches that can operate in unseen environments, classical approaches typically build a geometric map of the environment using depth sensors [12, 13, 14], and later a monocular RGB camera [15, 16, 17], while simultaneously localizing the robot relative to its growing map.”
BackgroundThe dynamic window approach to collision avoidance
Fox, D., Burgard, W., & Thrun, S. (1997) · IEEE Robotics & Automation Magazine · doi:10.1109/100.580977 · Cited by 3,771 · cited 1×
“Navigation has been studied in robotics literature for over three decades [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13].”
BackgroundA fast marching level set method for monotonically advancing fronts.
Sethian, J. A. (1996) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.93.4.1591 · Cited by 3,221 · cited 1×
“Planner Given a long-term goal output by the frontier or semantic exploration policy, we used the fast marching method (112) as in (63) to plan a path and the first low-level action along this path deterministically.”
Method
Research works printed but not linked: 16
- H. P. Moravec Obstacle avoidance and navigation in the real world by a seeing robot rover thesis Stanford University Palo Alto CA (1980).
- S. Thrun Robotic mapping: A survey in Exploring Artificial Intelligence in the New Millennium (Morgan Kaufmann Publishers Inc. 2002) pp. 1–35.
- U. Muller J. Ben E. Cosatto B. Flepp Y. Cun Off-road obstacle avoidance through end-to-end learning in Advances in Neural Information Processing Systems (MIT Press 2005) vol. 18.
- P. Mirowski R. Pascanu F. Viola H. Soyer A. Ballard A. Banino M. Denil R. Goroshin L. Sifre K. Kavukcuoglu D. Kumaran R. Hadsell Learning to navigate in complex environments ” paper presented at the 5th International Conference on Learning Representations Toulon France 24 to 26 2017 (ICLR 2017).
- A. Dosovitskiy V. Koltun “Learning to act by predicting the future ” paper presented at the 5th International Conference on Learning Representations Toulon France 24 to 26 2017 (ICLR 2017).
- P. Mirowski M. K. Grimes M. Malinowski K. M. Hermann K. Anderson D. Teplyashin K. Simonyan K. Kavukcuoglu A. Zisserman R. Hadsell Learning to navigate in cities without a map in Advances in Neural Information Processing Systems (Curran Associates Inc. 2018) vol. 31 pp. 2419–2430.
- E. Parisotto R. Salakhutdinov Neural map: Structured memory for deep reinforcement learning in International Conference on Learning Representations (ICLR) (2018).
- D. S. Chaplot E. Parisotto R. Salakhutdinov Active neural localization in International Conference on Learning Representations (ICLR) (2018).
- D. S. Chaplot D. P. Gandhi A. Gupta R. R. Salakhutdinov Object goal navigation using goal-oriented semantic exploration in Advances in Neural Information Processing Systems (Curran Associates Inc. 2020) vol. 33 p. 4247.
- M. Hahn D. S. Chaplot S. Tulsiani M. Mukadam J. M. Rehg A. Gupta No RL no simulation: Learning to navigate without navigating in Advances in Neural Information Processing Systems (Curran Associates Inc. 2021) vol. 34.
- X. Liu, H. Palacios, C. Muise, A planning based neural-symbolic approach for embodied instruction following. Interactions 9, 17 (2022).
- D. S. Chaplot M. Dalal S. Gupta J. Malik R. Salakhutdinov SEAL: Self-supervised Embodied Active Learning using exploration and 3D consistency in Advances in Neural Information Processing Systems (Curran Associates Inc. 2021).
- E. Kolve R. Mottaghi W. Han E. V. Bilt L. Weihs A. Herrasti M. Deitke K. Ehsani D. Gordon Y. Zhu A. Kembhavi A. Gupta A. Farhadi AI2-THOR: An interactive 3D environment for visual AI. arXiv:1712.05474 [cs.CV] (26 August 2022). · arXiv
- D. Kalashnikov A. Irpan P. Pastor J. Ibarz A. Herzog E. Jang D. Quillen E. Holly M. Kalakrishnan V. Vanhoucke S. Levine “Scalable deep reinforcement learning for vision-based robotic manipulation ” paper presented at the 2nd Annual Conference on Robot Learning (CoRL 2018) Zürich Switzerland 29 to 31 October 2018 pp. 651–673.
- A. Szot A. Clegg E. Undersander E. Wijmans Y. Zhao J. Turner N. Maestre M. Mukadam D. Chaplot O. Maksymets A. Gokaslan V. Vondrus S. Dharur F. Meier W. Galuba A. Chang Z. Kira V. Koltun J. Malik M. Savva D. Batra Habitat 2.0: Training home assistants to rearrange their habitat in Advances in Neural Information Processing Systems (Curran Associates Inc. 2021) vol. 34 p. 251.
- A. Murali T. Chen K. V. Alwala D. Gandhi L. Pinto S. Gupta A. Gupta PyRobot: An open-source robotics framework for research and benchmarking. arXiv:1906.08236 [cs.RO] (19 June 2019). · arXiv
Cited by
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., & Xia, F. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01370 · Cited by 281
“When applying these spatial concepts to reasoning, prior works often focus on explicit spatial scene memories [26, 27] or spatial scene graphs [29, 30, 59, 60].”
MethodDeep Reinforcement Learning for Robotics: A Survey of Real-World Successes
Tang, C., Abbatematteo, B., Hu, J., Chandra, R., Martín-Martín, R., & Stone, P. (2025) · Annual Review of Control, Robotics, and Autonomous Systems · doi:10.1146/annurev-control-030323-022510 · Cited by 177
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
Chen, B., Xu, Z., Kirmani, S., Ichter, B., Driess, D., Florence, P., Sadigh, D., Guibas, L., & Xia, F. (2024) · arXiv · doi:10.48550/arxiv.2401.12168 · Cited by 269
“When applying these spatial concepts to reasoning, prior works often focus on explicit spatial scene memories [26, 27] or spatial scene graphs [29, 30, 59, 60].”
MethodFoundation models in robotics: Applications, challenges, and the future
Firoozi, R., Tucker, J., Tian, S., Majumdar, A., Sun, J., Liu, W., Zhu, Y., Song, S., Kapoor, A., Hausman, K., et al. (2025) · The International Journal of Robotics Research · doi:10.1177/02783649241281508 · Cited by 238
“Semantic navigation [142] seeks to address this question by presenting an empirical analysis of semantic visual navigation methods.”
NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration
Sridhar, A., Shah, D., Glossop, C., & Levine, S. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610665 · Cited by 471
“However, policies trained in simulation tend to transfer poorly to real-world environments [16, 22], and our experiments reveal that even the best exploration policies trained on real-world data perform poorly in complex indoor and outdoor environments.”
Background
Metadata
- Authors
- Gervet, T., Chintala, S., Batra, D., Malik, J., & Chaplot, D. S.
- Affiliations
- Carnegie Mellon University, Pittsburgh, PA, USA.
- Meta (United States) (US)
- Georgia Institute of Technology (US)
- Venue
- Science Robotics
- Published
- 2023-06-28
- Volume
- 8
- Issue
- 79
- Pages
- eadf6991-eadf6991
- ISSN
- 2470-9476 (ISSN-L)
- OpenAlex
- W4382362889
- Cited by
- 125
- References
- 83
- Citations by year
- 2026: 17 · 2025: 50 · 2024: 44 · 2023: 14
- FWCI
- 11.06
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- Science Robotics (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotics and Sensor-Based Localization
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- mobile robots, object navigation, sim-to-real transfer, modular learning, end-to-end learning, exploration, domain gap, family environment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.