CoChat
Sign Up
Cited by 238

The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.

Foundation models in robotics: Applications, challenges, and the future

Firoozi, Roya, Tucker, Johnathan, Tian, Stephen, Majumdar, Anirudha, Sun, Jiankai, Liu, Weiyu, Zhu, Yuke, Song, Shuran, Kapoor, Ashish, Hausman, Karol, Ichter, Brian, Driess, Danny, Wu, Jiajun, Lu, Cewu, Schwager, Mac·2025·The International Journal of Robotics Research
DOI 10.1177/02783649241281508
Open this paper in CoChat — free account

Abstract

We survey applications of pretrained foundation models in robotics. Traditional deep learning models in robotics are trained on small datasets tailored for specific tasks, which limits their adaptability across diverse applications. In contrast, foundation models pretrained on internet-scale data appear to have superior generalization capabilities, and in some instances display an emergent ability to find zero-shot solutions to problems that are not present in the training data. Foundation models may hold the potential to enhance various components of the robot autonomy stack, from perception to decision-making and control. For example, large language models can generate code or provide common sense reasoning, while vision-language models enable open-vocabulary visual recognition. However, significant open research challenges remain, particularly around the scarcity of robot-relevant training data, safety guarantees and uncertainty quantification, and real-time execution. In this survey, we study recent papers that have used or built foundation models to solve robotics problems. We explore how foundation models contribute to improving robot capabilities in the domains of perception, decision-making, and control. We discuss the challenges hindering the adoption of foundation models in robot autonomy and provide opportunities and potential pathways for future advancements. The GitHub project corresponding to this paper can be found here: https://github.com/robotics-survey/Awesome-Robotics-Foundation-Models .

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1177/02783649241281508

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Integrating foundation models into real-world robotics remains challenging because even small errors in perception or reasoning can propagate through control loops, leading to irreversible outcomes.

    “However, integrating foundation models into real-world robotics remains challenging.Even small errors in perception or reasoning can propagate through control loops, leading to irreversible outcomes [7], [13].”

    From SwarmGPT: Combining Large Language Models With Safe Motion Planning for Drone Swarm Choreography (2025) · cites this paper as [13]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 215

  1. Anderson P Wu Q Teney D et al. (2018) Vision-and-language navigation: interpreting visually-grounded navigation instructions in real environments. In: CVPR pp. 3674–3683. IEEE. · DOI · Paper page
  2. Arandjelovic R Gronat P Torii A et al. (2016) NetVLAD: CNN architecture for weakly supervised place recognition. In: CVPR pp. 5297–5307. IEEE. · DOI
  3. Baker B Akkaya I Zhokov P et al. (2022) Video PreTraining (VPT): Learning to act by watching unlabeled online videos. In: NeurIPS. IEEE.
  4. 10.1109/TPAMI.2018.2798607 · DOI · Paper page
  5. Bangguo Y, Hamidreza K, Ming C (2023) LL3MVN: leveraging large language models for visual target navigation. arXiv preprint arXiv:2304.05501. · arXiv
  6. Bonatti R Vemprala S Ma S et al. (2023) PACT: perception-action causal transformer for autoregressive robotics pretraining. In: IROS. IEEE. · DOI
  7. Bousmalis K, Vezzani G, Rao D, et al. (2023) RoboCat: a self-improving foundation agent for robotic manipulation. arXiv preprint arXiv:2306.11706. · arXiv
  8. Box GE, Jenkins GM, Reinsel GC, et al. (2015) Time Series Analysis: Forecasting and Control. Hoboken, New Jersey: John Wiley & Sons.
  9. Brohan A, Brown N, Carbajal J, et al. (2022) Rt-1: robotics transformer for real-world control at scale arXiv Preprint arXiv:2212.06817. · arXiv
  10. Brohan A Brown N Carbajal J et al. (2023a) RT-1: robotics transformer for real-world control at scale. In: RSS. · DOI
  11. Brohan A Chebotar Y Finn C et al. (2023b) Do as I can not as I say: grounding language in robotic affordances. In: CoRL pp. 287–318. PMLR.
  12. Bucker A Figueredo L Haddadin S et al. (2023) LATTE: LAnguage trajectory TransformEr. In: ICRA pp. 7287–7294. IEEE. · DOI
  13. Cai F Koutsoukos X (2020) Real-time out-of-distribution detection in learning-enabled cyber-physical systems. In: ICCPS. IEEE. · DOI
  14. Chang AX, Funkhouser T, Guibas L, et al. (2015) ShapeNet: an information-rich 3D model repository. arXiv preprint arXiv:1512.03012. · arXiv
  15. Chen T Kornblith S Norouzi M et al. (2020) A simple framework for contrastive learning of visual representations. In: ICML.
  16. Chen M, Tworek J, Jun H, et al. (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. · arXiv
  17. Chen W, Hu S, Talak R, et al. (2022a) Leveraging large language models for robot 3D scene understanding. arXiv preprint arXiv:2209.05629. · arXiv
  18. Chen X Wang X Changpinyo S et al. (2022b) PaLI: a jointly-scaled multilingual language-image model. In NeurIPS. IEEE.
  19. Chen J, Li G, Kumar S, et al. (2023a) How to not train your dragon: training-free embodied object goal navigation with semantic frontiers. arXiv preprint arXiv:2305.16925. · arXiv
  20. 10.1126/scirobotics.adc9244 · DOI
  21. Chen X, Djolonga J, Padlewski P, et al. (2023c) PaLI-X: on scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565. · arXiv
  22. Chen Y, Arkin J, Zhang Y, et al. (2023d) AutoTAMP: autoregressive task and motion planning with llms as translators and checkers. arXiv preprint arXiv:2306.06531. · arXiv
  23. Chen Y, Gandhi R, Zhang Y, et al. (2023e) NL2TL: transforming natural languages to temporal logics using large language models. arXiv preprint arXiv:2305.07766. · arXiv
  24. Chen Z Kiami S Gupta A et al. (2023f) GenAug: retargeting behaviors to unseen situations via generative augmentation. In: RSS. · DOI
  25. Cheng HK Schwing AG (2022) XMem: long-term video object segmentation with an atkinson-shiffrin memory model. In: ECCV pp. 640–658. Springer. · DOI · Paper page
  26. Clarke S (2019) Robot Learning for Manipulation of Granular Materials Using Vision and Sound. Master’s Thesis. Carnegie Mellon University.
  27. Dai Z Yang Z Yang Y et al. (2019) Transformer-XL: attentive language models beyond a fixed-length context. In: ACL. · DOI · Paper page
  28. Damen D Doughty H Farinella GM et al. (2018) Scaling egocentric vision: the EPIC-KITCHENS dataset. In: ECCV. Springer. · DOI
  29. Dasari S Ebert F Tian S et al. (2019) RoboNet: large-scale multi-robot learning. In: CoRL.
  30. Dasgupta I Kaeser-Chen C Marino K et al. (2022) Collaborating with language models for embodied reasoning. In: Second Workshop on Language and Reinforcement Learning.
  31. Dehghani M Djolonga J Mustafa B et al. (2023) Scaling vision transformers to 22 billion parameters. In: ICML.
  32. DeTone D Malisiewicz T Rabinovich A (2018) Superpoint: self-supervised interest point detection and description. In: CVPR deep learning for visual SLAM workshop. IEEE. · DOI
  33. Ding W, Xu C, Arief M, et al. (2023) A survey on safety-critical driving scenario generation—a methodological perspective. IEEE ITSC 99: 1–19.
  34. Dong Q, Li L, Dai D, et al. (2022) A survey for in-context learning. arXiv preprint arXiv:2301.00234. · arXiv
  35. Dosovitskiy A Beyer L Kolesnikov A et al. (2021) An image is worth 16x16 words: transformers for image recognition at scale. In: ICLR.
  36. Du Y, Li J, Tang T, et al. (2023a) Zero-shot visual question answering with language model feedback. arXiv preprint arXiv:2305.17006. · arXiv
  37. Du Y Watkins O Wang Z et al. (2023b) Guiding pretraining in reinforcement learning with large language models. In: ICML pp. 8657–8677. PMLR.
  38. Du Y Yang M Dai B et al. (2023c) Learning universal policies via text-guided video generation. In: NeurIPS.
  39. Du Y, Yang M, Florence P, et al. (2023d) Video language planning. arXiv preprint arXiv:2310.10625. · arXiv
  40. Duan Y, Schulman J, Chen X, et al. (2016) RLˆ2: fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779. · arXiv
  41. Ehsani K Gupta T Hendrix R et al. (2024) Spoc: imitating shortest paths in simulation enables effective navigation and manipulation in the real world. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 16238–16250. IEEE. · DOI
  42. Engelbrecht HA Schiele G (2014) Transforming minecraft into a research platform. In: IEEE CCNC. IEEE. · DOI
  43. Fan L Wang G Jiang Y et al. (2022) MineDojo: building open-ended embodied agents with internet-scale knowledge. In: NeurIPS Datasets and Benchmarks Track.
  44. Fang HS Wang C Gou M et al. (2020) Graspnet-1billion: a large-scale benchmark for general object grasping. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 11444–11453. IEEE. · DOI · Paper page
  45. Farid A Veer S Majumdar A (2021) Task-driven out-of-distribution detection with statistical guarantees for robot learning. In: CoRL.
  46. Farid A, Snyder D, Ren AZ, et al. (2022a) Failure prediction with statistical guarantees for vision-based robot control. arXiv preprint arXiv:2202.05894. · arXiv
  47. Farid A Veer S Ivanovic B et al. (2022b) Task-relevant failure detection for trajectory predictors in autonomous vehicles. In: CoRL.
  48. Feng X, Luo Y, Wang Z, et al. (2023) ChessGPT: bridging policy learning and language modeling. arXiv preprint arXiv:2306.09200. · arXiv
  49. Fu L, Datta G, Huang H, et al. (2024) A touch, vision, and language dataset for multimodal alignment. arXiv preprint arXiv:2402.13232. · arXiv
  50. Gadre SY Wortsman M Ilharco G et al. (2023) CoWs on pasture: baselines and benchmarks for language-driven zero-shot object navigation. In: CVPR pp. 23171–23181. IEEE. · DOI
  51. Ganguli D, Lovitt L, Kernion J, et al. (2022) Red teaming language models to reduce harms: methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858. · arXiv
  52. Gao R Si Z Chang YY et al. (2022) Objectfolder 2.0: a multisensory object dataset for sim2real transfer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 10598–10608. IEEE. · DOI
  53. Gao J Sarkar B Xia F et al. (2023) Physically grounded vision-language models for robotic manipulation. In: CoRL. · DOI
  54. 10.1146/annurev-control-091420-084139 · DOI
  55. Goodwin W Havoutis I Posner I (2022a) You only look at one: category-level object representations for pose estimation from a single example. In: CoRL.
  56. Goodwin W Vaze S Havoutis I et al. (2022b) Zero-shot category-level object pose estimation. In: ECCV. · DOI
  57. Greenberg I Mannor S (2021) Detecting rewards deterioration in episodic reinforcement learning. In: ICML.
  58. Guzhov A Raue F Hees J et al. (2022) AudioCLIP: extending clip to image text and audio. In: ICASSP pp. 976–980. IEEE. · DOI · Paper page
  59. Ha H Song S (2022) Semantic abstraction: open-world 3D scene understanding from 2D vision-language models. In: CoRL.
  60. 10.1109/TPAMI.2022.3215150 · DOI
  61. He K, Zhang X, Ren S, et al. (2015) Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385. · arXiv
  62. He K Gkioxari G Dollár P et al. (2017) Mask R-CNN. In: ICCV pp. 2980–2988. DOI: 10.1109/ICCV.2017.322. · DOI · Paper page
  63. He X, Lin Z, Gong Y, et al. (2023) AnnoLLM: making large language models to be better crowdsourced annotators. arXiv preprint arXiv:2303.16854. · arXiv
  64. Ho J Jain A Abbeel P (2020) Denoising diffusion probabilistic models. In: NeurIPS.
  65. Hong Y, Zhen H, Chen P, et al. (2023) 3D-LLM: injecting the 3D world into large language models. arXiv preprint arXiv:2307.12981. · arXiv
  66. Hsu KC, Hu H, Fisac JF (2023a) The safety filter: a unified view of safety-critical control in autonomous systems. arXiv preprint arXiv:2309.05837. · arXiv
  67. 10.1016/j.artint.2022.103811 · DOI
  68. Hu Y, Xie Q, Jain V, et al. (2023b) Toward general-purpose robots via foundation models: a survey and meta-analysis. arXiv preprint arXiv:2312.08782. · arXiv
  69. Hu X, Wang Y, Fan L, et al. (2024) Semantic anything in 3d Gaussians. arXiv preprint arXiv:2401.17857. · arXiv
  70. Huang J Xie S Sun J et al. (2021) Learning a decision module by imitating driver’s control behaviors. In: CoRL pp. 1–10. PMLR.
  71. Huang W Abbeel P Pathak D et al. (2022a) Language models as zero-shot planners: extracting actionable knowledge for embodied agents. In: ICML.
  72. Huang C, Mees O, Zeng A, et al. (2023a) Audio visual language maps for robot navigation. arXiv preprint arXiv:2303.07522. · arXiv
  73. Huang W Wang C Zhang R et al. (2023c) VoxPoser: composable 3D value maps for robotic manipulation with language models. In: CoRL.
  74. Janner M Du Y Tenenbaum J et al. (2022) Planning with diffusion for flexible behavior synthesis. In: ICML.
  75. Jia Z, Liu F, Thumuluri V, et al. (2023) Chain-of-thought predictive control. arXiv preprint arXiv:2304.00776. · arXiv
  76. Jiang Y Gupta A Zhang Z et al. (2023) VIMA: general robot manipulation with multimodal prompts. In: ICML.
  77. 10.1109/LRA.2023.3272516 · DOI
  78. Karamcheti S Nair S Chen AS et al. (2023) Language-driven representation learning for robotics. In: RSS. · DOI
  79. Kerr J Kim CM Goldberg K et al. (2023) LERF: language embedded radiance fields. In: ICCV pp. 19729–19739. · DOI
  80. 10.1145/3505244 · DOI · Paper page
  81. Kobayashi S Matsumoto E Sitzmann V (2022) Decomposing NeRF for editing via feature field distillation. In: NeurIPS.
  82. Kumar A, Fu Z, Pathak D, et al. (2021) Rma: rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.04034. · arXiv
  83. Kusano KD, Beatty K, Schnelle S, et al. (2022) Collision avoidance testing of the waymo automated driving system. arXiv preprint arXiv:2212.08148. · arXiv
  84. Kwon M Xie SM Bullard K et al. (2023) Reward design with language models. In: ICLR.
  85. Li C Xia F Martín-Martín R et al. (2021) iGibson 2.0: object-centric simulation for robot learning of everyday household tasks. In: CoRL.
  86. Li B Weinberger KQ Belongie S et al. (2022a) Language-driven semantic segmentation. In: ICLR.
  87. Li C Zhang R Wong J et al. (2022b) BEHAVIOR-1K: a benchmark for embodied AI with 1 000 everyday activities and realistic simulation. In: CoRL.
  88. Li J Li D Xiong C et al. (2022c) BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In: ICML.
  89. Li LH Zhang P Zhang H et al. (2022d) Grounded language-image pre-training. In: CVPR pp. 10965–10975. · DOI
  90. Li Y Fan H Hu R et al. (2023) Scaling language-image pre-training via masking. In: CVPR. · DOI
  91. Liao G, Li J, Bao Z, et al. (2024) Clip-gs: clip-informed gaussian splatting for real-time and view-consistent 3d semantic understanding. arXiv preprint arXiv:2404.14249. · arXiv
  92. Lin J, Du Y, Watkins O, et al. (2023a) Learning to model the world with language. arXiv preprint arXiv:2308.01399. · arXiv
  93. Lin J, Tang J, Tang H, et al. (2023b) AWQ: activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978. · arXiv
  94. Lin K Agia C Migimatsu T et al. (2023c) Text2Motion: from natural language instructions to feasible plans. In: Special Issue: Large Language Models in Robotics. Autonomous Robots. · DOI
  95. Liu PJ Saleh M Pot E et al. (2018) Generating Wikipedia by summarizing long sequences. In: ICLR.
  96. Liu M, Zhu Y, Cai H, et al. (2023a) Partslip: low-shot part segmentation for 3d point clouds via pretrained image-language models. CVPR: 21736–21746.
  97. Liu S, Zeng Z, Ren T, et al. (2023c) Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499. · arXiv
  98. Liu W Du Y Hermans T et al. (2023d) StructDiffusion: language-guided creation of physically-valid structures using unseen objects. In: RSS. · DOI
  99. Luo R Zhao S Kuck J et al. (2022) Sample-efficient safety assurances using conformal prediction. In: WAFR. · DOI
  100. Lynch C Sermanet P (2021) Language conditioned imitation learning over unstructured data. In: Robotics: Science and Systems. · DOI · Paper page
  101. Lynch C Khansari M Xiao T et al. (2020) Learning latent plans from play. In: CoRL pp. 1113–1132. PMLR.
  102. Ma S Vemprala S Wang W et al. (2022) COMPASS: contrastive multimodal pretraining for autonomous systems. In: IROS pp. 1000–1007. IEEE. · DOI
  103. Ma YJ Kumar V Zhang A et al. (2023a) LIV: language-image representations and rewards for robotic control. In: ICML.
  104. Ma YJ Sodhani S Jayaraman D et al. (2023b) VIP: towards universal visual reward and representation via value-implicit pre-training. In:ICLR.
  105. Mahmoudieh P Pathak D Darrell T (2022) Zero-shot reward specification via grounded natural language. In ICML pp. 14743–14752. PMLR.
  106. Majumdar A Shrivastava A Lee S et al. (2020) Improving vision-and-language navigation with image-text pairs from the web. In: ECCV pp. 259–274. Springer. · DOI · Paper page
  107. Mandi Z, Bharadhwaj H, Moens V, et al. (2022) CACTI: a framework for scalable multi-task multi-scene visual imitation learning. arXiv preprint arXiv:2212.05711. · arXiv
  108. Margolis GB Agrawal P (2023) Walk these ways: tuning robot control for generalization with multiplicity of behavior. In: Conference on Robot Learning pp. 22–31. PMLR.
  109. Mees O Borja-Diaz J Burgard W (2023) Grounding language with visual affordances over unstructured data. In: ICRA pp. 11576–11582. IEEE. · DOI
  110. Minderer M Gritsenko A Stone A et al. (2022) Simple open-vocabulary object detection with vision transformers. In: ECCV pp. 728–755. Springer. · DOI
  111. Mirchandani S, Xia F, Florence P, et al. (2023) Large language models as general pattern machines. arXiv preprint arXiv:2307.04721. · arXiv
  112. Mu Y, Zhang Q, Hu M, et al. (2023) EmbodiedGPT: vision-language pre-training via embodied chain of thought. arXiv preprint arXiv:2305.15021. · arXiv
  113. Nair S Mitchell E Chen K et al. (2022a) Learning language-conditioned robot behavior from offline data and crowd-sourced annotation. In: CoRL pp. 1303–1315. PMLR.
  114. Nair S, Rajeswaran A, Kumar V, et al. (2022b) R3M: a universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601. · arXiv
  115. OpenAI (2023) GPT-4 technical report. arXiv preprint arXiv:2303.08774. · arXiv
  116. Padalkar A, Pooley A, Jain A, et al. (2023) Open X-Embodiment: robotic learning datasets and RT-X models. arXiv preprint arXiv:2310.08864. · arXiv
  117. Palo ND Byravan A Hasenclever L et al. (2023) Towards a unified agent with foundation models. In: Workshop on Reincarnating Reinforcement Learning at ICLR 2023.
  118. Park YJ, Wang H, Ardeshir S, et al. (2023b) Representation reliability and its impact on downstream tasks. arXiv preprint arXiv:2306.00206. · arXiv
  119. Puig X Ra K Boben M et al. (2018) VirtualHome: simulating household activities via programs. In: CVPR pp. 8494–8502. · DOI · Paper page
  120. Puig X, Undersander E, Szot A, et al. (2023) Habitat 3.0: a co-habitat for humans, avatars and robots. arXiv preprint arXiv:2310.13724. · arXiv
  121. Qin M, Li W, Zhou J, et al. (2023) Langsplat: 3d language Gaussian splatting. arXiv preprint arXiv:2312.16084. · arXiv
  122. 10.1109/JBHI.2023.3316750 · DOI
  123. Radford A, Wu J, Child R, et al. (2019) Language Models Are Unsupervised Multitask Learners. OpenAI Blog.
  124. Radford A Kim JW Hallacy C et al. (2021a) Learning transferable visual models from natural language supervision. In: Meila M Zhang T (eds) ICML Proceedings of Machine Learning Research Vol. 139 8748–8763. PMLR.
  125. Radford A Kim JW Hallacy C et al. (2021b) Learning transferable visual models from natural language supervision. In: ICML pp. 8748–8763.
  126. Radosavovic I Xiao T James S et al. (2023) Real-world robot learning with masked visual pre-training. In: CoRL pp. 416–426. PMLR.
  127. Ramesh A Pavlov M Goh G et al. (2021a) Zero-shot text-to-image generation. In: ICML pp. 8821–8831. PMLR.
  128. Ramesh A Pavlov M Goh G et al. (2021b) Zero-shot text-to-image generation. In: Meila M Zhang T (eds) Proceedings of the 38th International Conference on Machine Learning Proceedings of Machine Learning Research Vol. 139 pp. 8821–8831. PMLR.
  129. Rashid A Sharma S Kim CM et al. (2023) Language embedded radiance fields for zero-shot task-oriented grasping. In: 7th Annual Conference on Robot Learning (CoRL) pp. 178–200. PMLR.
  130. Reed S, Zolna K, Parisotto E, et al. (2022) A generalist agent. arXiv preprint arXiv:2205.06175. · arXiv
  131. Reid M, Yamada Y, Gu SS (2022) Can Wikipedia help offline reinforcement learning? arXiv preprint arXiv:2201.12122. · arXiv
  132. Ren AZ Dixit A Bodrova A et al. (2023) Robots that ask for help: uncertainty alignment for large language model planners. In: CoRL.
  133. Savva M Kadian A Maksymets O et al. (2019) Habitat: a platform for embodied AI research. In: ICCV. · DOI · Paper page
  134. Sennrich R Haddow B Birch A (2016) Neural machine translation of rare words with subword units. In: ACL. · DOI · Paper page
  135. Sermanet P, Ding T, Zhao J, et al. (2023) Robovqa: multimodal long-horizon reasoning for robotics. arXiv:2311.00899. · arXiv
  136. Shah R Kumar V (2021) RRL: resnet as representation for reinforcement learning. In: ICML.
  137. Shah D Osiński B Levine S et al. (2023a) LM-Nav: robotic navigation with large pre-trained models of language vision and action. In: CoRL pp. 492–504. PMLR.
  138. Shah D Sridhar A Dashora N et al. (2023b) ViNT: a foundation model for visual navigation. In: CoRL.
  139. Shah R Martín-Martín R Zhu Y (2023c) MUTEX: learning unified policies from multimodal task specifications. In: CoRL.
  140. Shen Q, Yang X, Wang X (2023a) Anything-3D: towards single-view anything reconstruction in the wild. arXiv preprint arXiv:2304.10261. · arXiv
  141. Shen W Yang G Yu A et al. (2023b) Distilled feature fields enable few-shot language-guided manipulation. In: 7th Annual Conference on Robot Learning (CoRL).
  142. Shen W Yang G Yu A et al. (2023c) Distilled feature fields enable few-shot manipulation. In: CoRL.
  143. Shridhar M Manuelli L Fox D (2022) CLIPort: what and where pathways for robotic manipulation. In: CoRL pp. 894–906. PMLR.
  144. Shridhar M Manuelli L Fox D (2023) Perceiver-Actor: a multi-task transformer for robotic manipulation. In: CoRL pp. 785–799. PMLR.
  145. Simeonov A Du Y Tagliasacchi A et al. (2022) Neural Descriptor Fields: SE(3)-equivariant object representations for manipulation. In: ICRA. · DOI
  146. Singh I Blukis V Mousavian A et al. (2023) ProgPrompt: generating situated robot task plans using large language models. In: ICRA pp. 11523–11530. IEEE. · DOI · Paper page
  147. Sinha R, Sharma A, Banerjee S, et al. (2022) A system-level view on out-of-distribution data in robotics. arXiv preprint arXiv:2212.14020. · arXiv
  148. Sohl-Dickstein J Weiss E Maheswaranathan N et al. (2015) Deep unsupervised learning using nonequilibrium thermodynamics. In: ICML.
  149. Sohn K (2016) Improved deep metric learning with multi-class n-pair loss objective. In: NeurIPS.
  150. Song Y Ermon S (2019) Generative modeling by estimating gradients of the data distribution. In: NeurIPS.
  151. Stone A, Xiao T, Lu Y, et al. (2023) Open-world object manipulation using pre-trained vision-language model. arXiv preprint arXiv:2303.00905. · arXiv
  152. Sun J Sun H Han T et al. (2021a) Neuro-symbolic program search for autonomous driving decision module design. In: CoRL pp. 21–30. PMLR.
  153. 10.1109/LRA.2021.3061397 · DOI
  154. 10.1109/LRA.2022.3150855 · DOI
  155. Sun J Kousik S Fridovich-Keil D et al. (2022b) Self-supervised traffic advisors: distributed multi-view traffic prediction for smart cities. In: IEEE ITSC. · DOI
  156. Sun M Yan W Abbeel P et al. (2022c) Quantifying uncertainty in foundation models via ensembles. In: NeurIPS Workshop on Robustness in Sequence Modeling.
  157. Sun J Jiang Y Qiu J et al. (2023a) Conformal prediction for uncertainty-aware planning with diffusion dynamics model. In: NeurIPS.
  158. Sun J, Kousik S, Fridovich-Keil D, et al. (2023b) Connected autonomous vehicle motion planning with video predictions from smart, self-supervised infrastructure. arXiv preprint arXiv:2309.07504. · arXiv
  159. Sun J, Qiu J, Zheng C, et al. (2023c) Aria-NeRF: multimodal egocentric view synthesis. arXiv preprint arXiv:2311.06455. · arXiv
  160. 10.1109/LRA.2023.3293308 · DOI
  161. Sun Y Ma S Madaan R et al. (2023f) SMART: self-supervised multi-task pretraining with control transformers. In: ICLR.
  162. Szot A Clegg A Undersander E et al. (2021) Habitat 2.0: training home assistants to rearrange their habitat. In: NeurIPS.
  163. Tarasov D Kurenkov V Kolesnikov S (2022) Prompts and pre-trained language models for offline reinforcement learning. In: ICLR Workshop on Generalizable Policy Learning in Physical World.
  164. Team OEL, Stooke A, Mahajan A, et al. (2021) Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808. · arXiv
  165. Team AA Bauer J Baumli K et al. (2023) Human-timescale adaptation in an open-ended task space. In: ICML.
  166. Tong S, Jones E, Steinhardt J (2023) Mass-producing failures of multimodal systems with language models. arXiv preprint arXiv:2306.12105. · arXiv
  167. Touvron H, Lavril T, Izacard G, et al. (2023b) LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. · arXiv
  168. Tschernezki V Laina I Larlus D et al. (2022) Neural Feature Fusion Fields: 3D distillation of self-supervised 2D image representations. In: 3DV pp. 443–453. IEEE. · DOI
  169. 10.1109/34.88573 · DOI
  170. Vaswani A Shazeer N Parmar N et al. (2017) Attention is all you need. In: NeurIPS.
  171. Villegas R Babaeizadeh M Kindermans PJ et al. (2023) Phenaki: variable length video generation from open domain textual description. In: ICLR.
  172. Vincent JA, Feldman AO, Schwager M (2023) Guarantees on robot system performance using stochastic simulation rollouts. arXiv preprint arXiv:2309.10874. · arXiv
  173. Vincent JA, Nishimura H, Itkina M, et al. (2024) How generalizable is my behavior cloning policy? a statistical approach to trustworthy performance evaluation. arXiv preprint arXiv:2405.05439. · arXiv
  174. Wang A, Singh A, Michael J, et al. (2018) GLUE: a multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. · arXiv
  175. Wang C Chai M He M et al. (2022) CLIP-NeRF: text-and-image driven manipulation of neural radiance fields. In: CVPR pp. 3835–3844. · DOI
  176. Wang C Fan L Sun J et al. (2023a) MimicPlay: long-horizon imitation learning by watching human play. In: CoRL.
  177. Wang S Saharia C Montgomery C et al. (2023c) Imagen editor and editbench: advancing and evaluating text-guided image inpainting. In: CVPR pp. 18359–18369. · DOI
  178. Webb N, Smith D, Ludwick C, et al. (2020) Waymo’s safety methodologies and safety readiness determinations. arXiv preprint arXiv:2011.00054. · arXiv
  179. Wei J, Wang X, Schuurmans D, et al. (2022) Chain-of-thought prompting elicits reasoning in large language models. NeurIPS 35: 24824–24837.
  180. 10.1007/s11704-023-2689-5 · DOI
  181. Xia F R Zamir A He ZY et al. (2018) Gibson Env: real-world perception for embodied agents. In: CVPR. · DOI · Paper page
  182. Xian Z, Gervet T, Xu Z, et al. (2023) Towards generalist robots: a promising paradigm via generative simulation. arXiv preprint arXiv:2305.10455. · arXiv
  183. Xiao T, Radosavovic I, Darrell T, et al. (2022) Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173. · arXiv
  184. Xiao T Chan H Sermanet P et al. (2023a) Robotic skill acquisition via instruction augmentation with vision-language models. In: RSS. · DOI
  185. Xiao X, Liu J, Wang Z, et al. (2023b) Robot learning in the era of foundation models: a survey. arXiv preprint arXiv:2311.14379. · arXiv
  186. Xue L, Gao M, Xing C, et al. (2022) ULIP: learning unified representation of language, image and point cloud for 3D understanding. arXiv preprint arXiv:2212.05171. · arXiv
  187. Xue L, Yu N, Zhang S, et al. (2023) ULIP-2: towards scalable multimodal pre-training for 3d understanding. arXiv preprint arXiv:2305.08275. · arXiv
  188. Yang J, Gao M, Li Z, et al. (2023a) Track anything: segment anything meets videos. arXiv preprint arXiv:2304.11968. · arXiv
  189. Yang F Feng C Chen Z et al. (2024) Binding touch to everything: learning unified multimodal tactile representations. In: Computer Vision and Pattern Recognition (CVPR). Springer. · DOI
  190. Yao L Huang R Hou L et al. (2022) FILIP: fine-grained interactive language-image pre-training. In: ICLR.
  191. Yao S Zhao J Yu D et al. (2023) ReAct: synergizing reasoning and acting in language models. In: ICLR.
  192. Ye J, Wang N, Wang X (2023a) FeatureNeRF: learning generalizable nerfs by distilling pre-trained vision foundation models. arXiv preprint arXiv:2303.12786. · arXiv
  193. Ye Y Li X Gupta A et al. (2023b) Affordance diffusion: synthesizing hand-object interactions. In: CVPR. · DOI
  194. Yenamandra S, Ramachandran A, Yadav K, et al. (2023) Homerobot: open vocabulary mobile manipulation. arXiv preprint arXiv:2306.11565. · arXiv
  195. Yi T, Mostafa D, Vinh QT, et al. (2023) UL2: unifying language learning paradigms. arXiv preprint arXiv:2205.05131. · arXiv
  196. Yoneda T, Fang J, Li P, et al. (2023) Statler: state-maintaining language models for embodied reasoning. arXiv preprint arXiv:2306.17840. · arXiv
  197. Yu X Tang L Rao Y et al. (2022) Point-BERT: pre-training 3D point cloud transformers with masked point modeling. In: CVPR pp. 19313–19322. Springer. · DOI
  198. Yu T, Xiao T, Stone A, et al. (2023) Scaling robot learning with semantically imagined experience. arXiv Preprint arXiv:2302.11550. · arXiv
  199. Yuying G Annabella M Li EL et al. (2023) Policy adaptation from foundation model feedback. In: CVPR.
  200. Zeng A Florence P Tompson J et al. (2020) Transporter networks: rearranging the visual world for robotic manipulation. In: CoRL.

200 of 215 shown

Cited by

Sorted by citations

Metadata

Authors
Firoozi, R., Tucker, J., Tian, S., Majumdar, A., Sun, J., Liu, W., Zhu, Y., Song, S., Kapoor, A., Hausman, K., Ichter, B., Driess, D., Wu, J., Lu, C., & Schwager, M.
Affiliations
  • Stanford University, Stanford, CA, USA
  • Princeton University, Princeton, NJ, USA
  • UT Austin, Austin, TX, USA
  • Scaled Foundations, Kirkland, WA, USA
  • Google DeepMind, Mountain View, CA, USA
  • Shanghai Jiao Tong University, Shanghai, China
Venue
The International Journal of Robotics Research
Publisher
SAGE Publications
Published
2024-09-25
Volume
44
Issue
5
Pages
701-739
ISSN
  • 0278-3649
  • 1741-3176
DOI
10.1177/02783649241281508
Cited by
238
References
39
Language
en
Licence
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Funders
  • Directorate for Engineering (2044149)
  • Defense Advanced Research Projects Agency (HR001120C0107)
  • Office of Naval Research (N00014-23-1-2148)
  • ASEE e-Fellows
  • NSF Graduate Research Fellowship

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review