CoChat
Sign Up
Cited by 42Reinforcement Learning in RoboticsRobotics and Sensor-Based LocalizationMultimodal Machine Learning Applications

Learning Navigational Visual Representations with Semantic Map Supervision

Open access
Hong, Yicong, Zhou, Yang, Zhang, Ruiyi, Dernoncourt, Franck, Bui, Trung, Gould, Stephen, Tan, Hao·2023·2023 IEEE/CVF International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv51070.2023.00284
Open this paper in CoChat — free accountOpen PDF

Abstract

Being able to perceive the semantics and the spatial structure of the environment is essential for visual navigation of a household robot. However, most existing works only employ visual backbones pre-trained either with independent images for classification or with self-supervised learning methods to adapt to the indoor navigation domain, neglecting the spatial relationships that are essential to the learning of navigation. Inspired by the behavior that humans naturally build semantically and spatially meaningful cognitive maps in their brains during navigation, in this paper, we propose a novel navigational-specific visual representation learning method by contrasting the agent’s egocentric views and semantic maps (Ego2-Map). We apply the visual transformer as the backbone encoder and train the model with data collected from the large-scale HabitatMatterport3D environments. Ego2-Map learning transfers the compact and rich information from a map, such as objects, structure and transition, to the agent’s egocentric representations for navigation. Experiments show that agents using our learned representations on object-goal navigation outperform recent visual pre-training methods. Moreover, our representations significantly improve vision-and-language navigation in continuous environments for both high-level and low-level action spaces, achieving new state-of-the-art results of 47% SR and 41% SPL on the test server.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccv51070.2023.00284

Cites

20 of 20 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Our method demonstrates obvious improvement compared to the previous method that applies very strong pre-trained visual representations.

    “Compared to the previous method, which applies very strong pre-trained visual representations [32, 81], our method still demonstrates obvious improvement”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [32]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 86

  1. Dong An, Zun Wang, Yangguang Li, Yi Wang, Yicong Hong, Yan Huang, Liang Wang, and Jing Shao. 1st place solutions for rxr-habitat vision-and-language navigation competition (cvpr 2022). arXiv preprint arXiv:2206.11610, 2022. 2 · arXiv
  2. Peter Anderson, Angel Chang, Devendra Singh Chap- lot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757, 2018. 6 · arXiv
  3. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3674- 3683, 2018. 1, 2, 5
  4. Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Olek- sandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects. arXiv preprint arXiv:2006.13171, 2020. 1, 2, 5, 8 · arXiv
  5. Arthur Bucker, Luis Figueredo, Sami Haddadin, Ashish Kapoor, Shuang Ma, and Rogerio Bonatti. Latte: Language trajectory transformer. arXiv preprint arXiv:2208.02918, 2022. 2 · arXiv
  6. Vincent Cartillier, Zhile Ren, Neha Jain, Stefan Lee, Irfan Essa, and Dhruv Batra. Semantic mapnet: Building allo- centric semantic maps and representations from egocentric views. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 964-972, 2021. 2, 3, 4, 14
  7. Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to ex- plore using active neural slam. In International Conference on Learning Representations, 2019. 1
  8. Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Ab- hinav Gupta, and Russ R Salakhutdinov. Object goal navi- gation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33:4247-4258, 2020. 2
  9. Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural topological slam for vi- sual navigation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 12875-12884, 2020. 2, 4
  10. Prithvijit Chattopadhyay, Judy Hoffman, Roozbeh Mot- taghi, and Aniruddha Kembhavi. Robustnav: Towards benchmarking robustness in embodied navigation. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 15691-15700, 2021. 3
  11. Peihao Chen, Dongyu Ji, Kunyang Lin, Runhao Zeng, Thomas Li, Mingkui Tan, and Chuang Gan. Weakly- supervised multi-granularity map learning for vision-and- language navigation. In Annual Conference on Neural In- formation Processing Systems, 2022. 2, 7, 8
  12. Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision-and-language navigation. Advances in Neural Infor- mation Processing Systems, 34:5834-5847, 2021. 2, 4
  13. Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Learning from unla- beled 3d environments for vision-and-language navigation. arXiv preprint arXiv:2208.11781, 2022. 2 · arXiv
  14. Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think global, act local: Dual-scale graph transformer for vision-and-language nav- igation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16537- 16547, 2022. 2
  15. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020. 7
  16. Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014. 8
  17. Yuchen Cui, Scott Niekum, Abhinav Gupta, Vikash Ku- mar, and Aravind Rajeswaran. Can foundation models per- form zero-shot task specification for robot manipulation? In Learning for Dynamics and Control Conference, pages 893-905. PMLR, 2022. 2
  18. Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, et al. Procthor: Large-scale embodied ai using procedural generation. arXiv preprint arXiv:2206.06994, 2022. 1, 2, 9 · arXiv
  19. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009. 7
  20. Saurabh Satish Desai and Stefan Lee. Auxiliary tasks for ef- ficient learning of point-goal navigation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pages 717-725, 2021. 3
  21. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In International Conference on Learning Representations, 2020. 4, 7
  22. Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned ob- jects: A high-quality dataset of 3d scanned household items. arXiv preprint arXiv:2204.11918, 2022. 8 · arXiv
  23. Yilun Du, Chuang Gan, and Phillip Isola. Curious represen- tation learning for embodied intelligence. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 10408-10417, 2021. 3
  24. Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi- task mid-level vision datasets from 3d scans. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 10786-10796, 2021. 8
  25. Russell A Epstein, Eva Zita Patai, Joshua B Julian, and Hugo J Spiers. The cognitive map in humans: spatial nav- igation and beyond. Nature neuroscience, 20(11):1504- 1513, 2017. 2
  26. Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. Clip on wheels: Zero- shot object navigation as object localization and explo- ration. arXiv preprint arXiv:2203.10421, 2022. 1, 2 · arXiv
  27. Georgios Georgakis, Bernadette Bucher, Karl Schmeck- peper, Siddharth Singh, and Kostas Daniilidis. Learning to map for active semantic goal navigation. In International Conference on Learning Representations, 2021. 2, 7
  28. Georgios Georgakis, Karl Schmeckpeper, Karan Wanchoo, Soham Dan, Eleni Miltsakaki, Dan Roth, and Kostas Dani- ilidis. Cross-modal map learning for vision and language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15460- 15470, 2022. 7, 8
  29. Meera Hahn, Devendra Singh Chaplot, Shubham Tulsiani, Mustafa Mukadam, James M Rehg, and Abhinav Gupta. No rl, no simulation: Learning to navigate without navigat- ing. Advances in Neural Information Processing Systems, 34:26661-26673, 2021. 4
  30. Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deploy- ment. In International Conference on Learning Represen- tations, 2020. 3
  31. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Gir- shick. Mask r-cnn. In Proceedings of the IEEE interna- tional conference on computer vision, pages 2961-2969, 2017. 1
  32. Yicong Hong, Cristian Rodriguez, Yuankai Qi, Qi Wu, and Stephen Gould. Language and visual entity relationship graph for agent navigation. Advances in Neural Informa- tion Processing Systems, 33:7685-7696, 2020. 1
  33. Yicong Hong, Zun Wang, Qi Wu, and Stephen Gould. Bridging the gap between learning in discrete and contin- uous environments for vision-and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 15439-15449, 2022. 5, 6, 7, 8
  34. Gabriel Ilharco, Vihan Jain, Alexander Ku, Eugene Ie, and Jason Baldridge. General evaluation for instruction con- ditioned navigation using dynamic time warping. arXiv preprint arXiv:1907.05446, 2019. 6 · arXiv
  35. Muhammad Zubair Irshad, Niluthpol Chowdhury Mithun, Zachary Seymour, Han-Pang Chiu, Supun Samarasek- era, and Rakesh Kumar. Sasra: Semantically-aware spatio-temporal reasoning agent for vision-and-language navigation in continuous environments. arXiv preprint arXiv:2108.11945, 2021. 2, 8 · arXiv
  36. Jindong Jiang, Lunan Zheng, Fei Luo, and Zhijun Zhang. Rednet: Residual encoder-decoder network for indoor rgb-d semantic segmentation. arXiv preprint arXiv:1806.01054, 2018. 1 · arXiv
  37. Jing Yu Koh, Harsh Agrawal, Dhruv Batra, Richard Tucker, Austin Waters, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson. Simple and effective synthesis of in- door 3d scenes. arXiv preprint arXiv:2204.02960, 2022. · arXiv
  38. Jing Yu Koh, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson. Pathdreamer: A world model for in- door navigation. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 14738- 14748, 2021. 2
  39. Jacob Krantz, Aaron Gokaslan, Dhruv Batra, Stefan Lee, and Oleksandr Maksymets. Waypoint models for instruction-guided navigation in continuous environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15162-15171, 2021. 8
  40. Jacob Krantz and Stefan Lee. Sim-2-sim transfer for vision- and-language navigation in continuous environments. In European Conference on Computer Vision, pages 588-603. Springer, 2022. 8
  41. Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Ba- tra, and Stefan Lee. Beyond the nav-graph: Vision-and- language navigation in continuous environments. In Euro- pean Conference on Computer Vision, 2020. 1, 2, 5, 8, 14
  42. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vi- sion, 123(1):32-73, 2017. 1
  43. Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-room: Multilingual vision-and-language navigation with dense spatiotempo- ral grounding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4392-4412, 2020. 1
  44. Chengshu Li, Fei Xia, Roberto Martín-Martín, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Elliott Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, et al. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. In Conference on Robot Learn- ing, pages 455-465. PMLR, 2022. 2
  45. Jialu Li and Mohit Bansal. Panogen: Text-conditioned panoramic environment generation for vision-and-language navigation. arXiv preprint arXiv:2305.19195, 2023. 2 · arXiv
  46. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014. 1
  47. Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learn- ing Representations, 2018. 5
  48. Haokuan Luo, Albert Yue, Zhang-Wei Hong, and Pulkit Agrawal. Stubborn: A strong baseline for indoor object navigation. arXiv preprint arXiv:2203.07359, 2022. 2 · arXiv
  49. Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter An- derson, Devi Parikh, and Dhruv Batra. Improving vision- and-language navigation with image-text pairs from the web. In European Conference on Computer Vision, pages 259-274. Springer, 2020. 2
  50. Oleksandr Maksymets, Vincent Cartillier, Aaron Gokaslan, Erik Wijmans, Wojciech Galuba, Stefan Lee, and Dhruv Batra. Thda: Treasure hunt data augmentation for se- mantic navigation. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 15374- 15383, 2021. 2
  51. Lina Mezghani, Sainbayar Sukhbaatar, Thibaut Lavril, Oleksandr Maksymets, Dhruv Batra, Piotr Bojanowski, and Karteek Alahari. Memory-augmented reinforce- ment learning for image-goal navigation. arXiv preprint arXiv:2101.05181, 2021. 1, 8 · arXiv
  52. Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint arXiv:1611.03673, 2016. 3 · arXiv
  53. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 4 · arXiv
  54. Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivas- tava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. Teach: Task-driven embodied agents that chat. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 36, pages 2017-2025, 2022. 2
  55. Simone Parisi, Aravind Rajeswaran, Senthil Purush- walkam, and Abhinav Gupta. The unsurprising effective- ness of pre-trained vision models for control. arXiv preprint arXiv:2203.03580, 2022. 2 · arXiv
  56. Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15942-15952, 2021. 1
  57. Yuankai Qi, Zizheng Pan, Yicong Hong, Ming-Hsuan Yang, Anton van den Hengel, and Qi Wu. The road to know-where: An object-and-room informed sequential bert for indoor vision-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 1655-1664, 2021. 3
  58. Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring ex- pression in real indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9982-9991, 2020. 2
  59. Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop: History-and-order aware pre- training for vision-and-language navigation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15418-15427, 2022. 2
  60. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021. 1, 2, 3, 5, 7, 8
  61. Santhosh Kumar Ramakrishnan, Devendra Singh Chap- lot, Ziad Al-Halah, Jitendra Malik, and Kristen Grauman. Poni: Potential functions for objectgoal navigation with interaction-free learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18890-18900, 2022. 2, 7
  62. Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wi- jmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew West- bury, Angel X Chang, et al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. In Thirty-fifth Conference on Neural Information Pro- cessing Systems Datasets and Benchmarks Track (Round 2), 2021. 2, 4, 8, 9, 14
  63. Santhosh K Ramakrishnan, Tushar Nagarajan, Ziad Al- Halah, and Kristen Grauman. Environment predictive cod- ing for embodied agents. arXiv preprint arXiv:2102.02337, 2021. 3 · arXiv
  64. Ram Ramrakhya, Eric Undersander, Dhruv Batra, and Ab- hishek Das. Habitat-web: Learning embodied object-search strategies from human demonstrations at scale. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5173-5183, 2022. 2, 8
  65. Sonia Raychaudhuri, Saim Wani, Shivansh Patel, Unnat Jain, and Angel Chang. Language-aligned waypoint (law) supervision for vision-and-language navigation in contin- uous environments. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Process- ing, pages 4018-4028, 2021. 8
  66. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Im- agenet large scale visual recognition challenge. Interna- tional journal of computer vision, 115(3):211-252, 2015.
  67. Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. In In- ternational Conference on Learning Representations, 2018. 3
  68. Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9339-9347, 2019. 2, 6, 7, 8, 9, 14
  69. Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. arXiv preprint arXiv:2207.04429, 2022. 2 · arXiv
  70. Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and-language tasks? In International Conference on Learning Represen- tations, 2021. 1, 2
  71. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, pages 10740-10749, 2020. 1, 2
  72. Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur- Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797, 2019. 8, 9 · arXiv
  73. Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems, 34:251-266, 2021. 2
  74. Ranxiao Frances Wang and Elizabeth S Spelke. Human spatial representation: Insights from animals. Trends in cognitive sciences, 6(9):376-382, 2002. 2
  75. Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint arXiv:1911.00357, 2019. 2, 6, 7, 8 · arXiv
  76. Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jiten- dra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 9068-9079, 2018. 1, 2, 7, 8, 9
  77. Karmesh Yadav, Ram Ramrakhya, Arjun Majumdar, Vincent-Pierre Berges, Sachit Kuhar, Dhruv Batra, Alexei Baevski, and Oleksandr Maksymets. Offline visual repre- sentation learning for embodied navigation. arXiv preprint arXiv:2204.13226, 2022. 2, 3, 8 · arXiv
  78. Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo net- works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1790- 1799, 2020. 8
  79. Joel Ye, Dhruv Batra, Abhishek Das, and Erik Wijmans. Auxiliary tasks and exploration enable objectnav. arXiv preprint arXiv:2104.04112, 2021. 1, 2, 3 · arXiv
  80. Joel Ye, Dhruv Batra, Erik Wijmans, and Abhishek Das. Auxiliary tasks speed up learning point goal navigation. In Conference on Robot Learning, pages 498-516. PMLR, 2021. 3
  81. Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 3712-3722, 2018. 8
  82. Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747, 2020. 4 · arXiv
  83. Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analy- sis and machine intelligence, 40(6):1452-1464, 2017. 1
  84. Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-language navigation with large lan- guage models. arXiv preprint arXiv:2305.16986, 2023. 2 · arXiv
  85. Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. Vision-language navigation with self-supervised auxiliary reasoning tasks. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 10012-10022, 2020. 4
  86. Minzhao Zhu, Binglei Zhao, and Tao Kong. Navigating to objects in unseen environments by distance prediction. arXiv preprint arXiv:2202.03735, 2022. 2 · arXiv

Cited by

Sorted by citations

Authors
Hong, Y., Zhou, Y., Zhang, R., Dernoncourt, F., Bui, T., Gould, S., & Tan, H.
Affiliations
  • Australian National University (AU)
  • Adobe Systems (United States) (US)
ORCID
Venue
2023 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2023-10-01
Pages
3032-3044
DOI
10.1109/iccv51070.2023.00284
OpenAlex
W4390872665
Cited by
42
References
20
Citations by year
2026: 10 · 2025: 19 · 2024: 12 · 2023: 1
FWCI
6.66
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
visual representation learning, semantic maps, visual navigation, home robots, egocentric view, vision transformer, object-goal navigation, vision-and-language navigation, spatial reasoning, habitat, Matterport3D, indoor navigation, cognitive maps

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review