CoChat
Sign Up
PreprintCited by 45Reinforcement Learning in RoboticsRobotics and Sensor-Based LocalizationRobot Manipulation and Learning

ViNT: A Foundation Model for Visual Navigation

Open access
Shah, Dhruv, Sridhar, Ajay, Dashora, Nitish, Stachowicz, Kyle, Black, Kevin, Hirose, Noriaki, Levine, Sergey·2023·arXiv
DOI 10.48550/arxiv.2306.14846arXiv 2306.14846
Open this paper in CoChat — free accountOpen PDF

Abstract

General-purpose pre-trained models ("foundation models") have enabled practitioners to produce generalizable solutions for individual machine learning problems with datasets that are significantly smaller than those required for learning from scratch. Such models are typically trained on large and diverse datasets with weak supervision, consuming much more training data than is available for any individual downstream application. In this paper, we describe the Visual Navigation Transformer (ViNT), a foundation model that aims to bring the success of general-purpose pre-trained models to vision-based robotic navigation. ViNT is trained with a general goal-reaching objective that can be used with any navigation dataset, and employs a flexible Transformer-based architecture to learn navigational affordances and enable efficient adaptation to a variety of downstream navigational tasks. ViNT is trained on a number of existing navigation datasets, comprising hundreds of hours of robotic navigation from a variety of different robotic platforms, and exhibits positive transfer, outperforming specialist models trained on singular datasets. ViNT can be augmented with diffusion-based subgoal proposals to explore novel environments, and can solve kilometer-scale navigation problems when equipped with long-range heuristics. ViNT can also be adapted to novel task specifications with a technique inspired by prompt-tuning, where the goal encoder is replaced by an encoding of another task modality (e.g., GPS waypoints or routing commands) embedded into the same space of goal tokens. This flexibility and ability to accommodate a variety of downstream problem domains establishes ViNT as an effective foundation model for mobile robotics. For videos, code, and model checkpoints, see our project page at https://visualnav-transformer.github.io.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.14846

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior transformer-based policy designs typically encode input images with large ResNet-style encoders and fuse the resulting image features with a comparatively small transformer.

    “Prior transformer-based policy designs typically encode input images with large ResNetstyle [32] encoders and fuse the resulting image features with a comparatively small transformer [10, 67, 81, 17, 98, 61, 83].”

    From Octo: An Open-Source Generalist Robot Policy (2024) · cites this paper as [81]

    Section 3 and the Goal fusion section describe ViNT encoding images into feature tokens and feeding them to a four-block transformer, supporting the general architecture attributed to it. But Appendix A identifies the image encoder as EfficientNet-B0, not a large ResNet-style encoder; the source also does not establish what prior designs typically use. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Prior architectures require re-initializing or re-training large components when adding or removing an image input or changing the task specification.

    “This is in contrast to prior architectures [10, 81], where adding or removing an image input or changing the task specification would require re-initializing or re-training large components of the pre-trained model.”

    From Octo: An Open-Source Generalist Robot Policy (2024) · cites this paper as [81]

    In “Method Performance Adaptation” (page unavailable), the paper says an early-fusion architecture would require learning its observation encoder entirely from scratch when adapting to a new goal modality. This supports the task-specification part of the claim, but our copy does not establish that adding or removing an image input would require re-initializing or re-training large components. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 45

Research works printed but not linked: 56

  1. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierar- chical image database. In Conference on Computer Vision and Pattern Recognition (CVPR), 2009. 1, 9
  2. J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  3. K. Grauman et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1
  4. A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022. 1 · arXiv
  5. M. Chen et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. 1 · arXiv
  6. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. International Conference on Machine Learning (ICML), 2020. 1, 9, 24
  7. D. Shah, B. Osiński, B. Ichter, and S. Levine. Robotic navigation with large pre-trained models of language, vision, and action. In Conference on Robot Learning (CoRL, 2022.
  8. X. Liu, Y. Li, C. Liang, and X. Li. The power of scale for parameter-efficient prompt tuning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
  9. B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. 1, 6
  10. C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In 2017 International Conference on Robotics and Automation (ICRA), 2017. 2
  11. S. Dasari, F. Ebert, et al. Robonet: Large-scale multi-robot learning. In Conference on Robot Learning (CoRL), 2020.
  12. F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. 2, 22
  13. P. Anderson, A. Shrivastava, J. Truong, A. Majumdar, D. Parikh, D. Batra, and S. Lee. Sim- to-real transfer for vision-and-language navigation. arXiv preprint arXiv:2011.03807, 2020. · arXiv
  14. J. Truong, A. Zitkovich, S. Chernova, D. Batra, T. Zhang, J. Tan, and W. Yu. Indoorsim-to- outdoorreal: Learning to navigate outdoors without any outdoor experience. arXiv preprint arXiv:2305.01098, 2023. 2, 5, 7 · arXiv
  15. A. Kadian, J. Truong, A. Gokaslan, A. Clegg, E. Wijmans, S. Lee, M. Savva, S. Chernova, and D. Batra. Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Perfor- mance? IEEE Robotics and Automation Letters, 2020. 2
  16. A. Loquercio, A. I. Maqueda, C. R. del Blanco, and D. Scaramuzza. Dronet: Learning to fly by driving. IEEE Robotics and Automation Letters, 2018. 2
  17. N. Hirose, D. Shah, A. Sridhar, and S. Levine. ExAug: Robot-Conditioned Navigation Poli- cies via Geometric Experience Augmentation. In International Conference on Robotics and Automation (ICRA), 2023.
  18. D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. GNM: A General Navigation Model to Drive Any Robot. In International Conference on Robotics and Automation (ICRA), 2023. 2, 4, 8, 9, 10, 23, 25
  19. Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: A Platform for Embod- ied AI Research. In International Conference on Computer Vision (ICCV), 2019. 2
  20. Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In International Confer- ence on Robotics and Automation (ICRA), 2017. 2
  21. F. Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy. End-to-End Driving Via Conditional Imitation Learning. In International Conference on Robotics and Automation (ICRA), 2018. 2, 6, 10, 20, 24
  22. N. Savinov, A. Dosovitskiy, and V. Koltun. Semi-Parametric Topological Memory for Naviga- tion. In International Conference on Learning Representations (ICLR), 2018. 2
  23. J. Bruce, N. Sunderhauf, P. Mirowski, R. Hadsell, and M. Milford. Learning deployable nav- igation policies at kilometer scale from a single traversal. In A. Billard, A. Dragan, J. Peters, and J. Morimoto, editors, 2nd Conference on Robot Learning, 2018.
  24. A. Faust, K. Oslund, O. Ramirez, A. Francis, L. Tapia, M. Fiser, and J. Davidson. Prm-rl: Long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning. In 2018 International Conference on Robotics and Automation (ICRA), 2018.
  25. X. Meng, N. Ratliff, Y. Xiang, and D. Fox. Scaling Local Control to Large-Scale Topological Navigation. In International Conference on Robotics and Automation (ICRA), 2020.
  26. D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine. ViNG: Learning Open- World Navigation with Visual Goals. In International Conference on Robotics and Automation (ICRA), 2021.
  27. N. Hirose, F. Xia, R. Martín-Martín, A. Sadeghian, and S. Savarese. Deep Visual MPC-Policy Learning for Navigation. IEEE Robotics and Automation Letters, 2019. 2, 22
  28. A. Brohan et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2023. 2, 3, 17, 18 · arXiv
  29. P. Anderson et al. On Evaluation of Embodied Navigation Agents. arXiv preprint arXiv:1807.06757, 2018. 3 · arXiv
  30. S. Wani, S. Patel, U. Jain, A. Chang, and M. Savva. Multion: Benchmarking semantic map memory using multi-object navigation. In Neural Information Processing Systems (NeurIPS), 2020. 3
  31. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, volume 30.
  32. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. 3
  33. M. Tan and Q. Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Con- ference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6105-6114. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr.press/v97/ tan19a.html. 3, 17
  34. K. Hartikainen, X. Geng, T. Haarnoja, and S. Levine. Dynamical Distance Learning for Semi- Supervised and Unsupervised Skill Discovery. In International Conference on Learning Rep- resentations (ICLR), 2020. 4
  35. D. Shah, B. Eysenbach, N. Rhinehart, and S. Levine. Rapid exploration for open-world nav- igation with latent goal models. In Conference on Robot Learning (CoRL), 2021. 5, 7, 22, 23
  36. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image syn- thesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022. 5
  37. J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Neural Information Processing Systems, 2020. 5, 18
  38. C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022. 5, 18
  39. D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.03378, 2023. 6 · arXiv
  40. Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03094, 2023. 6 · arXiv
  41. G. Kahn, A. Villaflor, B. Ding, P. Abbeel, and S. Levine. Self-Supervised Deep RL with Gen- eralized Computation Graphs for Robot Navigation. In International Conference on Robotics and Automation (ICRA), 2018. 7, 22, 23
  42. I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning (CoRL), 2022. 9, 25
  43. S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta. The unsurprising effectiveness of pre-trained vision models for control. arXiv preprint arXiv:2203.03580, 2022. 10 · arXiv
  44. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019. 19
  45. J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 19 · arXiv
  46. D. Kingma, T. Salimans, B. Poole, and J. Ho. Variational diffusion models. Neural Information Processing Systems (NeurIPS), 2021. 18
  47. J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 19 · arXiv
  48. C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gon- tijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Neural Information Processing Systems (NeurIPS), 2022. 19
  49. A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 20 · arXiv
  50. H. Karnan et al. Socially CompliAnt Navigation Dataset (SCAND): A Large-Scale Dataset Of Demonstrations For Social Navigation. IEEE Robotics and Automation Letters, 2022. 22
  51. A. Shaban, X. Meng, J. Lee, B. Boots, and D. Fox. Semantic terrain classification for off-road autonomous driving. In Conference on Robot Learning (CoRL), 2022. 22
  52. S. Triest et al. TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Models. In International Conference on Robotics and Automation (ICRA), 2022. 22
  53. A. Agha et al. Nebula: Quest for robotic autonomy in challenging environments; team costar at the darpa subterranean challenge. arXiv preprint arXiv:2103.11470, 2021. 22 · arXiv
  54. N. Hirose, D. Shah, A. Sridhar, and S. Levine. Sacson: Scalable autonomous control for social navigation. arXiv preprint arXiv:2306.01874, 2023. 22 · arXiv
  55. T. Niwa, S. Taguchi, and N. Hirose. Spatio-temporal graph localization networks for image- based navigation. In International Conference on Intelligent Robots and Systems (IROS), 2022. 22
  56. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 25

Authors
Shah, D., Sridhar, A., Dashora, N., Stachowicz, K., Black, K., Hirose, N., & Levine, S.
Affiliations
University of California, Berkeley (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2306.14846
arXiv
2306.14846
OpenAlex
W4382334278
Cited by
45
References
0
Citations by year
2025: 21 · 2024: 24
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2150826)
  • Nvidia
  • Army Research Office (W911NF-21-1-, W911NF-21-1-0097, W911NF, W911NF-17-2-0181)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
Keywords
visual navigation, foundation models, pre-trained vision models, transformer models, prompt tuning, task adaptation, mobile robotics, cross-dataset transfer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review