CoChat
Sign Up
Cited by 201Reinforcement Learning in RoboticsRobotics and Sensor-Based LocalizationRobot Manipulation and Learning

Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?

Open access
Kadian, Abhishek, Truong, Joanne, Gokaslan, Aaron, Clegg, Alexander, Wijmans, Erik, Lee, Stefan, Savva, Manolis, Chernova, Sonia +1 more·2020·IEEE Robotics and Automation Letters
DOI 10.1109/lra.2020.3013848
Open this paper in CoChat — free accountOpen PDF

Abstract

Does progress in simulation translate to progress on robots? If one method outperforms another in simulation, how likely is that trend to hold in reality on a robot? We examine this question for embodied PointGoal navigation - developing engineering tools and a research paradigm for evaluating a simulator by its sim2real predictivity. First, we develop Habitat-PyRobot Bridge (HaPy), a library for seamless execution of identical code on simulated agents and robots - transferring simulation-trained agents to a LoCoBot platform with a one-line code change. Second, we investigate the sim2real predictivity of Habitat-Sim M. Savva et al., for PointGoal navigation. We 3D-scan a physical lab space to create a virtualized replica, and run parallel tests of 9 different models in reality and simulation. We present a new metric called Sim-vs-Real Correlation Coefficient (SRCC) to quantify predictivity. We find that SRCC for Habitat as used for the CVPR19 challenge is low (0.18 for the success metric), suggesting that performance differences in this simulator-based challenge do not persist after physical deployment. This gap is largely due to AI agents learning to exploit simulator imperfections - abusing collision dynamics to `slide' along walls, leading to shortcuts through otherwise non-navigable space. Naturally, such exploits do not work in the real world. Our experiments show that it is possible to tune simulation parameters to improve sim2real predictivity (e.g. improving SRCCSuccfrom 0.18 to 0.844) - increasing confidence that in-simulation comparisons will translate to deployed systems in reality.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/lra.2020.3013848

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Learned agents that navigate without an explicit mapping module have shown strong performance on a variety of tasks.

    “Learned agents that navigate without an explicit mapping module (called 'map-free' or 'pixels-to-actions') have shown strong performance on a variety of tasks (Savva et al., 2019; Wijmans et al., 2020; Kadian et al., 2020; Chattopadhyay et al., 2021; Khandelwal et al., 2022; Partsey et al., 2022; Reed et al., 2022).”

    From Emergence of Maps in the Memories of Blind Navigation Agents (2023) · cites this paper as Kadian et al., 2020

    The Navigation Models section (p. 6) describes learned PointGoal agents with a visual encoder and policy network, rather than an explicit mapping module. The Results (p. 6) report success rates close to 1 in simulation, but this paper evaluates one navigation task, not strong performance across a variety of tasks; it also finds substantially more variable success in reality. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Neural-network models for navigation that do not contain explicit mapping modules still achieve remarkably high performance.

    “More recently, the machine learning commu- nity has produced a surprising phenomenon – neural-network models for navigation that curiously do not contain any explicit mapping modules but still achieve remarkably high performance (Savva et al., 2019; Wijmans et al., 2020; Kadian et al., 2020; Chattopadhyay et al., 2021; Khandelwal et al., 2022; Partsey et al., 2022; Reed et al., 2022).”

    From Emergence of Maps in the Memories of Blind Navigation Agents (2023) · cites this paper as Kadian et al., 2020

  • Large vision-language models like CLIP enable zero-shot generation of 3D textures and objects.

    “Subsequently, the advent of large vision-language models like CLIP [42] enables zero-shot generation of 3D textures and objects [14, 20, 28, 32, 33, 39].”

    From Holodeck: Language Guided Generation of 3D Embodied AI Environments (2024) · cites this paper as [20]

    The Evaluation Environment section (p. 5) describes reconstructing 3D meshes from camera scans, not generating textures or objects with a vision-language model. Searches for zero-shot 3D generation and CLIP found no passage in our copy addressing the attributed claim. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 31

  1. [1][1] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, et al. , 'Habitat: A Platform for Embodied AI Research,' in ICCV , 2019. 1, 2, 3, 4, 6 -
  2. [2][2] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, et al. , 'DD-PPO: Learning near-perfect pointgoal navigators from 2.5 billion frames,' in ICLR , 2020. 1, 2, 6 -
  3. [3][3] S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik, 'Cognitive mapping and planning for visual navigation,' in CVPR , 2017. 1 -
  4. [5][5] F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese, 'Gibson env: Real-world perception for embodied agents,' in CVPR , 2018. 1, 6 -
  5. [6][6] C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, et al. , 'Deepmind lab,' arXiv preprint arXiv:1612.03801 , 2016. 1, 6 - · arXiv
  6. [7][7] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, et al. , 'AI2THOR: An Interactive 3D Environment for Visual AI,' arXiv , 2017. 1, 3, 6 -
  7. [8][8] F. Sadeghi and S. Levine, 'CAD2RL: real single-image flight without a single real image,' in Robotics: Science and Systems XIII, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA, July 12-16, 2017 , 2017. 1, 3 -
  8. [9][9] M. Savva, A. X. Chang, A. Dosovitskiy, T. Funkhouser, and V. Koltun, 'MINOS: Multimodal indoor simulator for navigation in complex environments,' arXiv:1712.03931 , 2017. 1, 6 - · arXiv
  9. [10][10] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, et al. , 'Visionand-language navigation: Interpreting visually-grounded navigation instructions in real environments,' in CVPR , 2018. 1, 2, 5 -
  10. [11][11] A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, and D. Batra, 'Embodied Question Answering,' in CVPR , 2018. 1, 2 -
  11. [12][12] 'Visual Learning and Embodied Agents in Simulation Environments @ ECCV 2018,' https://eccv18-vlease.github.io/. 1 -
  12. [13][13] 'Habitat Challenge 2019 @ Habitat Embodied Agents Workshop. CVPR 2019,' https://aihabitat.org/challenge/2019/. 1, 4, 6 -
  13. [14][14] 'CARLA autonomous driving challenge @ CVPR 2019,' https:// carlachallenge.org/. 1 -
  14. [15][15] 'Robothor challenge @ CVPR 2020,' https://ai2thor.allenai.org/ robothor/challenge/. 1 -
  15. [16][16] J. Lehman, J. Clune, D. Misevic, C. Adami, L. Altenberg, et al. , 'The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities,' arXiv:1803.03453 , 2018. 2 - · arXiv
  16. [18][18] A. Murali, T. Chen, K. V. Alwala, D. Gandhi, L. Pinto, et al. , 'Pyrobot: An open-source robotics framework for research and benchmarking,' arXiv preprint arXiv:1906.08236 , 2019. 2, 3, 4, 7 - · arXiv
  17. [19][19] P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, et al. , 'On Evaluation of Embodied Navigation Agents,' arXiv preprint arXiv:1807.06757 , 2018. 2, 4, 6 - · arXiv
  18. [21][21] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, et al. , 'Target-driven visual navigation in indoor scenes using deep reinforcement learning,' ICRA , May 2017. 3 -
  19. [22][22] S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik, 'Cognitive mapping and planning for visual navigation,' CVPR , 2017. 3 -
  20. [24][24] E. Krotkov, D. Hackett, L. Jackel, M. Perschbacher, J. Pippine, et al. , 'The darpa robotics challenge finals: results and perspectives,' Journal of Field Robotics , vol. 34, no. 2, pp. 229-240, 2017. 3 -
  21. [25][25] D. Gandhi, L. Pinto, and A. Gupta, 'Learning to fly by crashing,' in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2017, pp. 3948-3955. 3 -
  22. [26][26] M. Müller, A. Dosovitskiy, B. Ghanem, and V. Koltun, 'Driving policy transfer via modularity and abstraction,' arXiv preprint arXiv:1804.09364 , 2018. 3 - · arXiv
  23. [27][27] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, et al. , 'Sim-toreal: Learning agile locomotion for quadruped robots,' arXiv preprint arXiv:1804.10332 , 2018. 3 - · arXiv
  24. [28][28] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, et al. , 'Closing the sim-to-real loop: Adapting simulation randomization with real world experience,' 05 2019, pp. 8973-8979. 3 -
  25. [29][29] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, et al. , 'Learning agile and dynamic motor skills for legged robots,' arXiv preprint arXiv:1901.08652 , 2019. 3 - · arXiv
  26. [30][30] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, et al. , 'Ros: an open-source robot operating system,' in ICRA workshop on open source software , vol. 3. Kobe, Japan, 2009, p. 5. 3 -
  27. [31][31] D. Mishkin, A. Dosovitskiy, and V. Koltun, 'Benchmarking classic and learned navigation in complex 3D environments,' arXiv:1901.10915 , 2019. 4 - · arXiv
  28. [32][32] N. Kojima and J. Deng, 'To learn or not to learn: Analyzing the role of learning for navigation in virtual environments,' arXiv:1907.11770 , 2019. 4 - · arXiv
  29. [33][33] S. Kohlbrecher, J. Meyer, O. von Stryk, and U. Klingauf, 'A flexible and scalable slam system with full 3d motion estimation,' in SSRR . IEEE, November 2011. 5 -
  30. [36][36] S. Hochreiter and J. Schmidhuber, 'Long short-term memory,' Neural Computation , vol. 9, no. 8, pp. 1735-1780, 1997. 6 -
  31. [37][37] Wofk, Diana and Ma, Fangchang and Yang, Tien-Ju and Karaman, Sertac and Sze, Vivienne, 'FastDepth: Fast Monocular Depth Estimation on Embedded Systems,' in ICRA , 2019. 6

Cited by

Sorted by citations

Authors
Kadian, A., Truong, J., Gokaslan, A., Clegg, A., Wijmans, E., Lee, S., Savva, M., Chernova, S., & Batra, D.
Affiliations
  • Meta (United States) (US)
  • Georgia Institute of Technology (US)
  • Oregon State University (US)
  • Simon Fraser University (CA)
ORCID
Venue
IEEE Robotics and Automation Letters
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2020-10
Volume
5
Issue
4
Pages
6670-6677
ISSN
  • 2377-3766 (ISSN-L)
  • 2377-3774
DOI
10.1109/lra.2020.3013848
OpenAlex
W3049043369
Cited by
201
References
5
Citations by year
2026: 24 · 2025: 40 · 2024: 44 · 2023: 32 · 2022: 28 · 2021: 24 · 2020: 9
FWCI
9.13
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Robotics and Automation Letters (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • NSF
  • Air Force Research Laboratory
  • Defense Advanced Research Projects Agency
  • ONR YIPs
  • ARO PECASE
  • Amazon Catalyst
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
Keywords
simulator evaluation, point-goal navigation, embodied agents, shortcut learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review