CoChat
Sign Up
PreprintCited by 177Reinforcement Learning in RoboticsRobot Manipulation and LearningRobotics and Sensor-Based Localization

ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects

Open access
Batra, Dhruv, Gokaslan, Aaron, Kembhavi, Aniruddha, Maksymets, Oleksandr, Mottaghi, Roozbeh, Savva, Manolis, Toshev, Alexander, Wijmans, Erik·2020·arXiv
DOI 10.48550/arxiv.2006.13171arXiv 2006.13171
Open this paper in CoChat — free accountOpen PDF

Abstract

We revisit the problem of Object-Goal Navigation (ObjectNav). In its simplest form, ObjectNav is defined as the task of navigating to an object, specified by its label, in an unexplored environment. In particular, the agent is initialized at a random location and pose in an environment and asked to find an instance of an object category, e.g., find a chair, by navigating to it. As the community begins to show increased interest in semantic goal specification for navigation tasks, a number of different often-inconsistent interpretations of this task are emerging. This document summarizes the consensus recommendations of this working group on ObjectNav. In particular, we make recommendations on subtle but important details of evaluation criteria (for measuring success when navigating towards a target object), the agent's embodiment parameters, and the characteristics of the environments within which the task is carried out. Finally, we provide a detailed description of the instantiation of these recommendations in challenges organized at the Embodied AI workshop at CVPR 2020 http://embodied-ai.org .

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.13171

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • HM3D and Gibson contain abundant, traversable, simulated indoor 3D scans collected from real-world buildings, which support learning of various visual navigation problems.

    “Both datasets contain abundant, traversable, and simulated indoor 3D scans collected from real-world buildings, which support the learning of various visual navigation problems [12, 44, 55, 70].”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [12]

    Section 2.3 (p. 4) describes Gibson as 572 scanned, virtualized spaces suitable for ObjectNav, supporting the Gibson portion of the claim. The paper discusses Matterport3D (MP3D), not HM3D, and does not establish that HM3D supports the claimed range of visual navigation problems. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Existing vision-and-language navigation datasets face a common data scarcity issue.

    “To tackle the common data scarcity issue in existing vision-and-language navigation datasets”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [12]

    The paper discusses ObjectNav datasets in Section 3 (p. 5) and mentions VLN only in connection with path-comparison metrics in Section 2.1.1 (p. 4). Searches for vision-and-language navigation dataset scarcity and limited training data found no passage addressing the claimed common data-scarcity issue. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 20

  1. [1][1] CVPR 2020 Embodied AI Workshop. https:// embodied-ai.org/ , 2020. 1, 2 -
  2. [2][2] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On Evaluation of Embodied Navigation Agents. arXiv preprint arXiv:1807.06757 , 2018. 1, 2 - · arXiv
  3. [3][3] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2019. 1, 2, 4, 5, 6 -
  4. [4][4] Abhishek Kadian, Joanne Truong, Aaron Gokaslan, Alexander Clegg, Erik Wijmans, Stefan Lee, Manolis Savva, Sonia Chernova, and Dhruv Batra. Are we making real progress in simulated environments? measuring the sim2real gap in embodied visual navigation. arXiv preprint arXiv:1912.06321 , 2019. 1, 6, 7 - · arXiv
  5. [5][5] Habitat Challenge 2019 @ Habitat Embodied Agents Workshop. CVPR 2019. https://aihabitat.org/challenge/ 2019/ . 1, 2, 6 -
  6. [6][6] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied Question Answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. 2 -
  7. [7][7] Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. IQA: Visual question answering in interactive environments. In CVPR , 2018. 2 -
  8. [9][9] Fei Xia, William B Shen, Chengshu Li, Priya Kasimbeg, Micael Tchapmi, Alexander Toshev, Roberto Martín-Martín, and Silvio Savarese. Interactive gibson: A benchmark for interactive navigation in cluttered environments. RA-Letters , 2020. 4 -
  9. [10][10] Vihan Jain, Gabriel Magalhaes, Alex Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. Stay on the path: Instruction fidelity in vision-and-language navigation. In ACL , 2019. 4 -
  10. [11][11] Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. DD-PPO: Learning near-perfect pointgoal navigators from 2.5 billion frames. ICLR , 2020. 4 -
  11. [12][12] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740-755. Springer, 2014. 4 -
  12. [13][13] Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801 , 2016. 4, 7 - · arXiv
  13. [14][14] Marek Wydmuch, Michał Kempka, and Wojciech Ja´ skowski. Vizdoom competitions: Playing doom from pixels. IEEE Transactions on Games , 2018. 4 -
  14. [16][16] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3D: Learning from RGBD Data in Indoor Environments. International Conference on 3D Vision (3DV) , 2017. MatterPort3D dataset license available at: http://kaldir.vc.in.tum.de/matterport/ MP\_TOS.pdf . 4, 5 -
  15. [17][17] Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 9068-9079, 2018. Gibson dataset license agreement available at https://storage.googleapis.com/gibson\_ material/Agreement%20GDS%2006-04-18.pdf . 5, 7 -
  16. [18][18] Iro Armeni, Zhi-Yang He, JunYoung Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3D scene graph: A structure for unified semantics, 3d space, and camera. In Proceedings of the IEEE International Conference on Computer Vision , pages 5664-5673, 2019. 5 -
  17. [19][19] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv , 2017. 5, 7 -
  18. [20][20] Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation using scene priors. In ICLR , 2019. 5 -
  19. [21][21] Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Learning to learn how to learn: Self-adaptive visual navigation using meta-learning. In CVPR , 2019. 5 -
  20. [22][22] Manolis Savva, Angel X. Chang, Alexey Dosovitskiy, Thomas Funkhouser, and Vladlen Koltun. MINOS: Multimodal indoor simulator for navigation in complex environments. arXiv:1712.03931 , 2017. 7 · arXiv

Cited by

Sorted by citations

Authors
Batra, D., Gokaslan, A., Kembhavi, A., Maksymets, O., Mottaghi, R., Savva, M., Toshev, A., & Wijmans, E.
Affiliations
  • Georgia Institute of Technology (US)
  • Meta (United States) (US)
  • Allen Institute for Artificial Intelligence (US)
  • Simon Fraser University (CA)
  • Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2006.13171
arXiv
2006.13171
OpenAlex
W3036497058
Cited by
177
References
2
Citations by year
2026: 19 · 2025: 19 · 2024: 38 · 2023: 43 · 2022: 27 · 2021: 28 · 2020: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
Keywords
object-goal navigation, embodied agents, evaluation criteria, robot navigation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review