CoChat
Sign Up
PreprintCited by 321Reinforcement Learning in RoboticsRobotics and Sensor-Based LocalizationSpatial Cognition and Navigation

Object Goal Navigation using Goal-Oriented Semantic Exploration

Open access
Chaplot, Devendra Singh, Gandhi, Dhiraj, Gupta, Abhinav, Salakhutdinov, Ruslan·2020·arXiv
DOI 10.48550/arxiv.2007.00643arXiv 2007.00643
Open this paper in CoChat — free accountOpen PDF

Abstract

This work studies the problem of object goal navigation which involves navigating to an instance of the given object category in unseen environments. End-to-end learning-based navigation methods struggle at this task as they are ineffective at exploration and long-term planning. We propose a modular system called, `Goal-Oriented Semantic Exploration' which builds an episodic semantic map and uses it to explore the environment efficiently based on the goal object category. Empirical results in visually realistic simulation environments show that the proposed model outperforms a wide range of baselines including end-to-end learning-based methods as well as modular map-based methods and led to the winning entry of the CVPR-2020 Habitat ObjectNav Challenge. Ablation analysis indicates that the proposed model learns semantic priors of the relative arrangement of objects in a scene, and uses them to explore efficiently. Domain-agnostic module design allow us to transfer our model to a mobile robot platform and achieve similar performance for object goal navigation in the real-world.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2007.00643

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most existing methods process observations in a reactive and stateless manner.

    “Most existing methods, whether based on end-to-end reinforcement learning18–20 or modular pipelines with powerful MLLMs21,22, process observations in a reactive and stateless manner.”

    From Brain-inspired spatial intelligence for embodied agents (2026) · cites this paper as [18]

    The Introduction (p. 1) describes end-to-end methods that use recurrent networks to build episodic memory and a modular system that maintains memory with explicit maps—contrary to calling these methods stateless. The paper does not address MLLM-based pipelines. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 39

  1. [1][1] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757 , 2018. - · arXiv
  2. [2][2] Iro Armeni, Zhi-Yang He, JunYoung Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In Proceedings of the IEEE International Conference on Computer Vision , pages 5664-5673, 2019. -
  3. [3][3] Sean L Bowman, Nikolay Atanasov, Kostas Daniilidis, and George J Pappas. Probabilistic data association for semantic slam. In 2017 IEEE international conference on robotics and automation (ICRA) , pages 1722-1729. IEEE, 2017. -
  4. [4][4] John Canny. The complexity of robot motion planning . MIT press, 1988. -
  5. [6][6] Devendra Singh Chaplot and Guillaume Lample. Arnold: An autonomous agent to play fps games. In Thirty-First AAAI Conference on Artificial Intelligence , 2017. -
  6. [7][7] Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. arXiv preprint arXiv:1706.07230 , 2017. - · arXiv
  7. [8][8] Devendra Singh Chaplot, Emilio Parisotto, and Ruslan Salakhutdinov. Active neural localization. ICLR , 2018. -
  8. [9][9] Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to explore using active neural slam. In International Conference on Learning Representations (ICLR) , 2020. -
  9. [11][11] Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural topological slam for visual navigation. In CVPR , 2020. -
  10. [12][12] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 , 2017. - · arXiv
  11. [13][13] J. Fuentes-Pacheco, J. Ruiz-Ascencio, and J. M. Rendón-Mancha. Visual simultaneous localization and mapping: a survey. Artificial Intelligence Review , 2015. -
  12. [14][14] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estimation with left-right consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 270-279, 2017. -
  13. [15][15] Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering in interactive environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4089-4098, 2018. -
  14. [16][16] Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2616-2625, 2017. -
  15. [17][17] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In 2017 IEEE International Conference on Computer Vision (ICCV) , pages 2980-2988, 2017. -
  16. [19][19] Joao F Henriques and Andrea Vedaldi. Mapnet: An allocentric spatial memory for mapping environments. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8476-8484, 2018. -
  17. [20][20] Peter Henry, Michael Krainin, Evan Herbst, Xiaofeng Ren, and Dieter Fox. Rgb-d mapping: Using depth cameras for dense 3d modeling of indoor environments. In Experimental robotics , pages 477-491. Springer, 2014. -
  18. [21][21] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojtek Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning in a simulated 3d world. arXiv preprint arXiv:1706.06551 , 2017. - · arXiv
  19. [22][22] Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, and Andrew Fitzgibbon. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. UIST , 2011. -
  20. [25][25] Guillaume Lample and Devendra Singh Chaplot. Playing FPS games with deep reinforcement learning. In Thirty-First AAAI Conference on Artificial Intelligence , 2017. -
  21. [27][27] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014. -
  22. [29][29] Lingni Ma, Jörg Stückler, Christian Kerl, and Daniel Cremers. Multi-view deep learning for consistent semantic mapping with rgb-d cameras. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 598-605. IEEE, 2017. -
  23. [30][30] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments. ICLR , 2017. -
  24. [31][31] Arsalan Mousavian, Alexander Toshev, Marek Fišer, Jana Košecká, Ayzaan Wahid, and James Davidson. Visual representations for semantic target driven navigation. In 2019 International Conference on Robotics and Automation (ICRA) , pages 8846-8852. IEEE, 2019. -
  25. [32][32] Adithyavairavan Murali, Tao Chen, Kalyan Vasudev Alwala, Dhiraj Gandhi, Lerrel Pinto, Saurabh Gupta, and Abhinav Gupta. Pyrobot: An open-source robotics framework for research and benchmarking. arXiv preprint arXiv:1906.08236 , 2019. - · arXiv
  26. [34][34] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. NIPS 2017 Autodiff Workshop , 2017. -
  27. [35][35] Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. In International Conference on Learning Representations (ICLR) , 2018. -
  28. [36][36] Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. In ICLR , 2019. -
  29. [37][37] Manolis Savva, Angel X. Chang, Alexey Dosovitskiy, Thomas Funkhouser, and Vladlen Koltun. MINOS: Multimodal indoor simulator for navigation in complex environments. arXiv:1712.03931 , 2017. - · arXiv
  30. [38][38] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE International Conference on Computer Vision , pages 9339-9347, 2019. -
  31. [40][40] James A Sethian. A fast marching level set method for monotonically advancing fronts. Proceedings of the National Academy of Sciences , 93(4):1591-1595, 1996. -
  32. [41][41] Noah Snavely, Steven M Seitz, and Richard Szeliski. Modeling the world from internet photo collections. International journal of computer vision , 80(2):189-210, 2008. -
  33. [42][42] Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Decentralized distributed ppo: Solving pointgoal navigation. arXiv preprint arXiv:1911.00357 , 2019. - · arXiv
  34. [43][43] Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Learning to learn how to learn: Self-adaptive visual navigation using meta-learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6750-6759, 2019. -
  35. [44][44] Yi Wu, Yuxin Wu, Aviv Tamar, Stuart Russell, Georgia Gkioxari, and Yuandong Tian. Learning and planning with a semantic model. arXiv preprint arXiv:1809.10842 , 2018. - · arXiv
  36. [45][45] Fei Xia, Amir R. Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson Env: real-world perception for embodied agents. In Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on . IEEE, 2018 Gibson dataset license agreement available at https://storage.googleapis.com/gibson\_ material/Agreement%20GDS%2006-04-18.pdf. -
  37. [47][47] Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation using scene priors. arXiv preprint arXiv:1810.06543 , 2018. - · arXiv
  38. [48][48] Liang Zhang, Leqi Wei, Peiyi Shen, Wei Wei, Guangming Zhu, and Juan Song. Semantic slam based on object detection and improved octomap. IEEE Access , 6:75545-75559, 2018. -
  39. [49][49] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA) , pages 3357-3364. IEEE, 2017.

Cited by

Sorted by citations

Authors
Chaplot, D. S., Gandhi, D., Gupta, A., & Salakhutdinov, R.
Affiliations
  • Carnegie Mellon University (US)
  • Meta (Israel) (IL)
ORCID
Gupta, Abhinav
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
33
Pages
4247-4258
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2007.00643
arXiv
2007.00643
OpenAlex
W3102804550
Cited by
321
References
8
Citations by year
2026: 3 · 2025: 33 · 2024: 78 · 2023: 89 · 2022: 58 · 2021: 55 · 2020: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Funders
  • Nvidia
  • Defense Advanced Research Projects Agency
  • Intelligence Advanced Research Projects Activity
  • Stanford Bio-X
  • Office of Naval Research
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
  • Physical Sciences › Engineering › Automotive Engineering › Spatial Cognition and Navigation
Keywords
object-goal navigation, semantic exploration, goal-directed navigation, unknown environments, semantic priors, object arrangement, long-term planning, mobile robot navigation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review