Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?
Open accessAbstract
We present the largest and most comprehensive empirical study of pre-trained visual representations (PVRs) or visual 'foundation models' for Embodied AI. First, we curate CortexBench, consisting of 17 different tasks spanning locomotion, navigation, dexterous, and mobile manipulation. Next, we systematically evaluate existing PVRs and find that none are universally dominant. To study the effect of pre-training data size and diversity, we combine over 4,000 hours of egocentric videos from 7 different sources (over 4.3M images) and ImageNet to train different-sized vision transformers using Masked Auto-Encoding (MAE) on slices of this data. Contrary to inferences from prior work, we find that scaling dataset size and diversity does not improve performance universally (but does so on average). Our largest model, named VC-1, outperforms all prior PVRs on average but does not universally dominate either. Next, we show that task- or domain-specific adaptation of VC-1 leads to substantial gains, with VC-1 (adapted) achieving competitive or superior performance than the best known results on all of the benchmarks in CortexBench. Finally, we present real-world hardware experiments, in which VC-1 and VC-1 (adapted) outperform the strongest pre-existing PVR. Overall, this paper presents no new techniques but a rigorous systematic evaluation, a broad set of findings about PVRs (that in some cases, refute those made in narrow domains in prior work), and open-sourced code and models (that required over 10,000 GPU-hours to train) for the benefit of the research community.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Octo achieved an average success rate of 72% across six finetuning evaluation setups, compared to 20% for ResNet+Transformer Scratch and 15% for VC-1.
“| Octo (Ours) | 70% | 75% | 50% | 60% | 100% | 80% | 72% |”
From Octo: An Open-Source Generalist Robot Policy (2024) · cites this paper as [57]
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 7× · see citing works
“We choose the number of epochs per run such that the number of model updates remain constant across all runs and match the number of model updates taken by MAE on the ImageNet dataset.”
MethodInfluentialAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · cited 12× · see citing works
“VC-1 is superior to the second-best model ( Ego4D+MN ViT-L, Table 4 row 9), which has an average rank of 3.1.”
MethodInfluentialMomentum Contrast for Unsupervised Visual Representation LearningPreprint
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. B. (2019) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1911.05722 · Cited by 16,250 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 7× · see citing works
“3 However, curiously, the kinds of PVRs that are locally-dominant in CORTEXBENCH differ significantly in the size and type of pre-training datasets: CLIP (Radford et al., 2021) was pretrained on 400M image-text pairs from the web; MVP (Radosavovic et al.”
BackgroundInfluentialA simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · cited 1× · see citing works
“These algorithms use contrastive [21, 22], distillation-based [19, 20], or reconstructive [18, 23] objectives for training.”
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · cited 10× · see citing works
“Since VC-1 was trained with MAE (He et al., 2021), it captures features that are generally useful for reconstructing images.”
BackgroundMethodInfluentialFukushima, K. (1980) · Biological Cybernetics · doi:10.1007/bf00344251 · Cited by 5,759 · cited 2× · see citing works
“In this work, we ask the same question that Fukushima [1, 2] asked nearly 50 years ago – how do we design an artificial visual cortex , the module in a computational system that (together with a policy) enables an agent to convert camera input into actions?”
BackgroundEmerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 1× · see citing works
“End-to-end (E2E) fine-tuning with a task-specific loss function can in-principle capture both of the aforementioned benefits of adaptation, and is widely used in computer vision literature (He et al., 2020; Caron et al., 2021; He et al., 2021; Baevski et al., 2022b).”
MethodChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1× · see citing works
“We use a broad definition of adaptation [55], which, in the context of large pre-trained foundation models, can take several forms from simple prompting [56], to selectively updating some or all weights of the backbone [5, 35, 57].”
MethodOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., & Joulin, A. (2020) · arXiv · doi:10.48550/arxiv.2006.09882 · Cited by 2,499 · cited 2× · see citing works
“The last few years have seen increasing interest in the self-supervised learning (SSL) of visual representations (He et al., 2021; Caron et al., 2020; Baevski et al., 2022b; Chen et al., 2020; 2021).”
BackgroundAn Empirical Study of Training Self-Supervised Vision Transformers
Chen, X., Xie, S., & He, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00950 · Cited by 1,648 · see citing works
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., & Farhadi, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989381 · Cited by 1,533 · cited 1× · see citing works
“We consider two semantic navigation tasks in Habitat: image-goal navigation (ImageNav) (Zhu et al., 2017) and object-goal navigation (ObjectNav) (Batra et al.”
MethodLearning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J. D., Todorov, E., & Levine, S. (2017) · Robotics: Science and Systems Conference · doi:10.48550/arxiv.1709.10087 · Cited by 1,522 · cited 2×
“, 2020), dexterous manipulation (Rajeswaran et al., 2018), multifinger coordinated manipulation (Wüthrich et al.”
BackgroundSelf-citationThe “Something Something” Video Database for Learning and Evaluating Visual Common Sense
Goyal, R., Kahou, S. E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.622 · Cited by 1,471 · cited 2× · see citing works
“One cluster of datasets – Ego4D [49], 100 Days of Hands (100DOH) [50], Something-Something v2 (SS-V2) [51], and Epic Kitchens [52] – contains videos of people manipulating objects and is comparable to the datasets used in MVP [8].”
MethodBEiT: BERT Pre-Training of Image Transformers
Bao, H., Dong, L., Piao, S., & Wei, F. (2021) · arXiv · doi:10.48550/arxiv.2106.08254 · Cited by 1,392 · see citing works
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and LanguagePreprint
Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., & Auli, M. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2202.03555 · Cited by 1,303 · cited 2× · see citing works
“End-to-end (E2E) fine-tuning with a task-specific loss function can in-principle capture both of the aforementioned benefits of adaptation, and is widely used in computer vision literature [18, 20, 58, 59].”
MethodHabitat: A Platform for Embodied AI Research
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00943 · Cited by 1,237 · cited 2× · see citing works
“The dataset is comprised of 80 training, 20 validation, and 20 testing scenes and utilizes the Habitat simulator [15, 65] and HM3D [68] environments.”
MethodSelf-citationFine-Tuning can Distort Pretrained Features and Underperform Out-of-DistributionPreprint
Kumar, A., Raghunathan, A., Jones, R., Ma, T.-Y., & Liang, P. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2202.10054 · Cited by 1,110 · cited 1× · see citing works
“, 2022), to selectively updating some or all weights of the backbone (Kumar et al., 2022; Hansen et al., 2022a; Yadav et al., 2023).”
BackgroundGibson Env: Real-World Perception for Embodied AgentsPreprint
Xia, F., Zamir, A., He, Z.-Y., Sax, A., Malik, J., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1808.10654 · Cited by 1,076 · see citing works
Self-citationZhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891 · cited 1× · see citing works
“Several works have showed that scaling model and dataset size improves performance on vision tasks like image classification (Zhai et al., 2022; Tian et al., 2021; Goyal et al., 2021).”
BackgroundCognitron: A self-organizing multilayered neural network
Fukushima, K. (1975) · Biological cybernetics · doi:10.1007/bf00342633 · Cited by 639 · cited 2× · see citing works
“In this work, we ask the same question that Fukushima [1, 2] asked nearly 50 years ago – how do we design an artificial visual cortex , the module in a computational system that (together with a policy) enables an agent to convert camera input into actions?”
BackgroundEgo4D: Around the World in 3,000 Hours of Egocentric Video
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01842 · Cited by 630 · cited 1× · see citing works
“These datasets include Ego4D [22], 100 Days of Hands (100DOH) [23], Something-Something v2 (SS-V2) [24], and Epic Kitchens [25].”
MethodSelf-citationVIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-TrainingPreprint
Ma, Y., Sodhani, S., Jayaraman, D., Bastani, O., Kumar, V., & Zhang, A. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2210.00030 · Cited by 591 · cited 3× · see citing works
“Inspired by the advancements in self-supervised learning, recent work has incorporated visual representation learning into the training pipelines for EAI agents [3–9].”
BackgroundMethodInfluentialSelf-citationLast Layer Re-Training is Sufficient for Robustness to Spurious CorrelationsPreprint
Kirichenko, P., Izmailov, P., & Wilson, A. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2204.02937 · Cited by 533 · see citing works
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. D. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1801.00690 · Cited by 497 · cited 2× · see citing works
“We curate C ORTEX B ENCH , a benchmark for evaluating PVRs, consisting of 17 tasks spanning low-level locomotion [10], table-top manipulation [11], dexterous manipulation [12], multi-finger coordinated manipulation [13], indoor visual navigation [14], and mobile manipulation [15].”
BackgroundMethodMeta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
Yu, T., Quillen, D., He, Z., Julian, R., Narayan, A., Shively, H., Bellathur, A., Hausman, K., Finn, C., & Levine, S. (2019) · arXiv · doi:10.48550/arxiv.1910.10897 · Cited by 450 · cited 2× · see citing works
“MetaWorld (MW) (Yu et al., 2020) is a collection of tasks in which an agent commands a Sawyer robot arm to manipulate objects in a tabletop environment.”
BackgroundUnderstanding Human Hands in Contact at Internet ScalePreprint
Shan, D., Geng, J., Shu, M., & Fouhey, D. F. (2020) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.2006.06669 · Cited by 433 · cited 1× · see citing works
“, 2022), 100 Days of Hands (100DOH) (Shan et al., 2020), Something-Something v2 (SS-V2) (Goyal et al.”
Real-World Robot Learning with Masked Visual Pre-trainingPreprint
Radosavovic, I., Xiao, T.-T., James, S., Abbeel, P., Malik, J., & Darrell, T. (2022) · Conference on Robot Learning · doi:10.48550/arxiv.2210.03109 · Cited by 389 · cited 5× · see citing works
“, 2021) was pretrained on 400M image-text pairs from the web; MVP (Radosavovic et al., 2022) on 4.”
MethodInfluentialSelf-citationSurgical Fine-Tuning Improves Adaptation to Distribution ShiftsPreprint
Lee, Y., Chen, A. S., Tajwar, F., Kumar, A., Yao, H., Liang, P., & Finn, C. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2210.11466 · Cited by 326 · see citing works
LIV: Language-Image Representations and Rewards for Robotic Control
Ma, Y., Liang, W., Som, V., Kumar, V., Zhang, A., Bastani, O., & Jayaraman, D. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2306.00958 · Cited by 312 · cited 2×
“[28] focuses on self-supervised representation learning for goal-conditioned value functions using language-aligned videos.”
BackgroundSelf-citationScaling Egocentric Vision: The EPIC-KITCHENS Dataset
Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2018) · arXiv · doi:10.48550/arxiv.1804.02748 · Cited by 274 · cited 1× · see citing works
“One cluster of datasets – Ego4D [49], 100 Days of Hands (100DOH) [50], Something-Something v2 (SS-V2) [51], and Epic Kitchens [52] – contains videos of people manipulating objects and is comparable to the datasets used in MVP [8].”
MethodDD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion FramesPreprint
Wijmans, E., Kadian, A., Morcos, A. S., Lee, S., Essa, I. A., Parikh, D., Savva, M., & Batra, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00357 · Cited by 253 · see citing works
Self-citationThe Unsurprising Effectiveness of Pre-Trained Vision Models for ControlPreprint
Parisi, S., Rajeswaran, A., Purushwalkam, S., & Gupta, A. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2203.03580 · Cited by 251 · cited 5× · see citing works
“, 2022), DMControl from (Parisi et al., 2022), ImageNav from (Yadav et al.”
BackgroundMethodResultInfluentialSelf-citationFinetune like you pretrain: Improved finetuning of zero-shot vision models
Goyal, S., Kumar, A., Garg, S., Kolter, Z., & Raghunathan, A. (2022) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52729.2023.01853 · Cited by 243 · see citing works
Language-Driven Representation Learning for Robotics
Karamcheti, S., Nair, S., Chen, A. S., Kollar, T., Finn, C., Sadigh, D., & Liang, P. (2023) · Robotics: Science and Systems Conference · doi:10.48550/arxiv.2302.12766 · Cited by 240 · cited 2×
“[27] trains a ViT with a masked encoding objective on pairs of image frames and text.”
BackgroundObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects
Batra, D., Gokaslan, A., Kembhavi, A., Maksymets, O., Mottaghi, R., Savva, M., Toshev, A., & Wijmans, E. (2020) · arXiv · doi:10.48550/arxiv.2006.13171 · Cited by 177 · cited 2× · see citing works
“We consider two tasks in Habitat: image-goal navigation ( ImageNav ) [44] and object-goal navigation ( ObjectNav ) [45].”
BackgroundMethodSelf-citationSimple but Effective: CLIP Embeddings for Embodied AI
Khandelwal, A., Weihs, L., Mottaghi, R., & Kembhavi, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01441 · Cited by 172 · cited 2× · see citing works
“Inspired by the advancements in self-supervised learning, recent work has incorporated visual representation learning into the training pipelines for EAI agents (Parisi et al., 2022; Nair et al., 2022; Radosavovic et al., 2022; Ma et al., 2022; Khandelwal et al., 2022; Yadav et al., 2022b; 2023).”
BackgroundMethodRobot Learning in Homes: Improving Generalization and Reducing Dataset BiasPreprint
Gupta, A., Murali, A., Gandhi, D., & Pinto, L. (2018) · Neural Information Processing Systems · doi:10.48550/arxiv.1807.07049 · Cited by 170 · see citing works
Baevski, A., Babu, A., Hsu, W.-N., & Auli, M. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2212.07525 · Cited by 166 · cited 7×
“(2020); Baevski et al. (2022b); Chen et al. (2020; 2021). These algorithms use contrastive Chen et al. (2020; 2021), distillation-based Caron et al. (2020); Baevski et al. (2022b), or reconstructive Bao et al. (2021); He et al. (2021) objectives for training. Recently, a flurry of works have proposed using the vision…”
BackgroundMethodInfluentialProcTHOR: Large-Scale Embodied AI Using Procedural Generation
Deitke, M., VanderBilt, E., Herrasti, A., Weihs, L., Salvador, J., Ehsani, K., Han, W., Kolve, E., Farhadi, A., Kembhavi, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.06994 · Cited by 155 · cited 2× · see citing works
“Thesolutions,basedonautoregressivelanguagemodelinginGPT[47,48,4]and maskedautoencoding in BERT[14],areconceptuallysimple:theyremoveaportionofthedataandlearntopredicttheremovedcontent.”
BackgroundHabitat 2.0: Training home assistants to rearrange their habitat
Szot, A., Clegg, A., Undersander, E., Wijmans, E., Zhao, Y., Turner, J., Maestre, N., Mukadam, M., Chaplot, D., Maksymets, O., et al. (2021) · arXiv · doi:10.48550/arxiv.2106.14405 · Cited by 143 · cited 6× · see citing works
“We curate C ORTEX B ENCH , a benchmark for evaluating PVRs, consisting of 17 tasks spanning low-level locomotion [10], table-top manipulation [11], dexterous manipulation [12], multi-finger coordinated manipulation [13], indoor visual navigation [14], and mobile manipulation [15].”
BackgroundMethodInfluentialSelf-citationSelf-supervised Pretraining of Visual Features in the Wild
Goyal, P., Caron, M., Lefaudeux, B., Xu, M., Wang, P., Pai, V., Singh, M., Liptchinsky, V., Misra, I., Joulin, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.01988 · Cited by 141 · cited 1× · see citing works
“Several works have showed that scaling model and dataset size improves performance on vision tasks like image classification (Zhai et al., 2022; Tian et al., 2021; Goyal et al., 2021).”
BackgroundLife Ascending: The Ten Great Inventions of Evolution
Lane, N. (2009) · Choice Reviews Online · doi:10.5860/choice.47-4400 · Cited by 130 · see citing works
Offline Visual Representation Learning for Embodied NavigationPreprint
Yadav, K., Ramrakhya, R., Majumdar, A., Berges, V.-P., Kuhar, S., Batra, D., Baevski, A., & Maksymets, O. (2022) · arXiv.org · doi:10.48550/arxiv.2204.13226 · Cited by 118 · see citing works
Self-citationHabitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI
Ramakrishnan, S. K., Gokaslan, A., Wijmans, E., Maksymets, O., Clegg, A., Turner, J., Undersander, E., Galuba, W., Westbury, A., Chang, A. X., et al. (2021) · arXiv · doi:10.48550/arxiv.2109.08238 · Cited by 106 · cited 1× · see citing works
“The dataset is comprised of 80 training, 20 validation, and 20 testing scenes and utilizes the Habitat simulator [15, 65] and HM3D [68] environments.”
MethodSelf-citationOVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavPreprint
Yadav, K., Majumdar, A., Ramrakhya, R., Yokoyama, N., Baevski, A., Kira, Z., Maksymets, O., & Batra, D. (2023) · arXiv.org · doi:10.48550/arxiv.2303.07798 · Cited by 103 · cited 1× · see citing works
“, 2022), to selectively updating some or all weights of the backbone (Kumar et al., 2022; Hansen et al., 2022a; Yadav et al., 2023).”
BackgroundSelf-citationThe Surprising Effectiveness of Representation Learning for Visual Imitation
Pari, J., Shafiullah, N. M., Arunachalam, S. P., & Pinto, L. (2021) · arXiv · doi:10.48550/arxiv.2112.01511 · Cited by 94 · cited 1× · see citing works
“By comparison, [41] find simple”
Habitat-Matterport 3D Semantics Dataset
Yadav, K., Ramrakhya, R., Ramakrishnan, S. K., Gervet, T., Turner, J., Gokaslan, A., Maestre, N., Chang, A. X., Batra, D., Savva, M., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00477 · Cited by 92 · cited 3× · see citing works
“We present an evaluation of object navigation ( ObjectNav ) using the HM3D-S EM dataset [61].”
MethodSelf-citationMulti-skill Mobile Manipulation for Object RearrangementPreprint
Gu, J., Chaplot, D. S., Su, H., & Malik, J. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2209.02778 · Cited by 91 · cited 1× · see citing works
“, 2021) includes a set of mobile manipulation tasks in which an agent controls a Fetch robot with a 7-DoF arm, mobile base (Gu et al., 2022), and suction gripper to rearrange objects in apartment scenes.”
BackgroundSelf-citationHabitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scale
Ramrakhya, R., Undersander, E., Batra, D., & Das, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00511 · Cited by 90 · see citing works
Self-citationOn Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline
Hansen, N., Yuan, Z.-C., Ze, Y., Mu, T.-Z., Rajeswaran, A., Su, H., Xu, H., & Wang, X.-L. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2212.05749 · Cited by 86
Self-citationTriFinger: An Open-Source Robot for Learning DexterityPreprint
Wüthrich, M., Widmaier, F., Grimminger, F., Akpo, J., Joshi, S., Agrawal, V., Hammoud, B., Khadiv, M., Bogdanovic, M., Berenz, V., et al. (2020) · Conference on Robot Learning · doi:10.48550/arxiv.2008.03596 · Cited by 86 · see citing works
MoDem: Accelerating Visual Model-Based Reinforcement Learning with DemonstrationsPreprint
Hansen, N., Lin, Y.-X., Su, H., Wang, X.-L., Kumar, V., & Rajeswaran, A. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2212.05698 · Cited by 79 · see citing works
Self-citationRoboHive: A Unified Framework for Robot LearningPreprint
Kumar, V., Shah, R. M., Zhou, G.-Y., Moens, V., Caggiano, V., Vakil, J., Gupta, A., & Rajeswaran, A. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2310.06828 · Cited by 73 · cited 1× · see citing works
“We use Robohive [73] as the middleware to interface with all sensors and robots.”
MethodSelf-citationPIRLNav: Pretraining with Imitation and RL Finetuning for OBJECTNAV
Ramrakhya, R., Batra, D., Wijmans, E., & Das, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01716 · Cited by 71 · see citing works
Self-citationRetrospectives on the Embodied AI WorkshopPreprint
Deitke, M., Batra, D., Bisk, Y., Campari, T., Chang, A. X., Chaplot, D. S., Chen, C., Pérez-D'Arpino, C., Ehsani, K., Farhadi, A., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2210.06849 · Cited by 62 · see citing works
Self-citationMemory-Augmented Reinforcement Learning for Image-Goal Navigation
Mezghan, L., Sukhbaatar, S., Lavril, T., Maksymets, O., Batra, D., Bojanowski, P., & Alahari, K. (2022) · 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981090 · Cited by 61 · cited 1× · see citing works
“Our study conducts ImageNav experiments using the standard dataset presented in [64].”
MethodSelf-citationZero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation
Al-Halah, Z., Ramakrishnan, S. K., & Grauman, K. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01652 · Cited by 58 · cited 1× · see citing works
“We train agents with the reward functions presented in [67] utilizing the following settings: success weighting c s = 5 .”
MethodDivide and Contrast: Self-supervised Learning from Uncurated Data
Tian, Y., Hénaff, O. J., & van den Oord, A. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00991 · Cited by 57 · cited 1× · see citing works
“Several works have showed that scaling model and dataset size improves performance on vision tasks like image classification (Zhai et al., 2022; Tian et al., 2021; Goyal et al., 2021).”
BackgroundOpen-World Object Manipulation using Pre-trained Vision-Language Models
Stone, A., Xiao, T., Lu, Y., Gopalakrishnan, K., Lee, K.-H., Vuong, Q., Wohlhart, P., Kirmani, S., Zitkovich, B., Xia, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.00905 · Cited by 33 · cited 2× · see citing works
“Suchmethodsincludecontrastivelearningwithdataaugmentation[25,26,27,28],dynamicsprediction[29,30],bi-simulation[31],temporalorgoaldistance[32,33],ordomainspecificinformation[34]. from 7 existing EAI…”
Pave the Way to Grasp Anything: Transferring Foundation Models for Universal Pick-Place Robots
Yang, J.-G., Tan, W., Jin, C., Liu, B., Fu, J., Song, R.-H., & Wang, L.-M. (2023) · arXiv.org · doi:10.48550/arxiv.2306.05716 · Cited by 30 · cited 2×
“Suchmethodsincludecontrastivelearningwithdataaugmentation[25,26,27,28],dynamicsprediction[29,30],bi-simulation[31],temporalorgoaldistance[32,33],ordomainspecificinformation[34]. from 7 existing EAI…”
VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied RearrangementPreprint
Wijmans, E., Essa, I., & Batra, D. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2210.05064 · Cited by 27 · see citing works
Self-citationSchreiber, M. (1990) · doi:10.1097/00004424-199010000-00017 · Cited by 20
CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learning
Mandi, Z., Bharadhwaj, H., Moens, V., Song, S., Rajeswaran, A., & Kumar, V. (2022) · arXiv · doi:10.48550/arxiv.2212.05711 · Cited by 19 · cited 1× · see citing works
“• The close-drawer and toaster plunge tasks are similar to the Cacti setup [72].”
MethodSelf-citationInstruction-Following Agents with Multimodal TransformerPreprint
Liu, H., Lee, L., Lee, K., & Abbeel, P. (2022) · arXiv.org · doi:10.48550/arxiv.2210.13431 · Cited by 17 · cited 2× · see citing works
“In [31], a multimodal transformer is pretrained on web-scale image-and-text data, and then used with a transformer-based policy for table-top manipulation tasks.”
MethodSelf-citationMasked Contrastive Representation Learning
Yao, Y., Desai, N., & Palaniswami, M. (2022) · arXiv.org · doi:10.48550/arxiv.2211.06012 · Cited by 8 · cited 1×
“, 2021) with masked image modeling (He et al., 2021; Baevski et al., 2022a; Yao et al., 2022), which among other benefits reduces the computation time required for pre-training.”
BackgroundR3M: A Universal Visual Representation for Robot Manipulation
Nair, S., Rajeswaran, A., Kumar, V., Finn, C., & Gupta, A. (2022) · arXiv · doi:10.48550/arxiv.2203.12601 · cited 14×
“WefindthatR3Menablesdataefficientimitationlearningacrossseveralsimulatedandreal-worldrobotmanipulationtasks.”
MethodInfluentialSelf-citation
Flagged first, then by citations
15 of the 83 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models
Zhao, Q., Lu, Y., Kim, M. J., Fu, Z.-P., Zhang, Z.-Y., Wu, Y.-C., Li, Z.-S., Ma, Q.-L., Han, S., Finn, C., et al. (2025) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.00166 · Cited by 665 · see citing works
“Some approaches incorporate VLMs [3, 44, 58] into end-to-end trainable policies by using them as pretrained backbone for better visuo-language representation [10, 26, 38, 43, 53].”
An Embodied Generalist Agent in 3D WorldPreprint
Huang, J., Yong, S.-L., Ma, X., Linghu, X., Li, P.-H., Wang, Y., Li, Q., Zhu, S.-C., Jia, B., & Huang, S.-Y. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2311.12871 · Cited by 475 · see citing works
HumanPlus: Humanoid Shadowing and Imitation from HumansPreprint
Fu, Z.-P., Zhao, Q., Wu, Q., Wetzstein, G., & Finn, C. (2024) · Conference on Robot Learning · doi:10.48550/arxiv.2406.10454 · Cited by 433 · see citing works
“Human data has been used extensively for robot learning, including for pre-training visual or intermediate representations or tasks [8, 28, 50, 53, 66, 79, 86, 99] leveraging Internet-scale data [18, 26, 27], and for imitation learning on in-domain human data [3, 12, 43, 47”
MethodA Survey on Vision--Language--Action Models for Embodied AI
Ma, Y., Song, Z., Zhuang, Y.-Z., Hao, J., & King, I. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3650584 · Cited by 431
“Pixel-level information contains rich details—including segmentation masks, object positions, and depth estimation—that are generally more useful for robot manipulation tasks requiring high precision, as demonstrated by VC-1’s comparison [32].”
Self-citationVideo Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
Hu, Y.-C., Guo, Y., Wang, P., Chen, X.-Y., Wang, Y.-J., Zhang, J.-K., Sreenath, K., Lu, C., & Chen, J.-Y. (2024) · International Conference on Machine Learning · doi:10.48550/arxiv.2412.14803 · Cited by 426
“No-tably, methods like R3M (Nair et al., 2022), vip (Ma et al., 2022), VC-1 (Majumdar et al., 2023), and Voltron (Karam-cheti et al., 2023) have specifically focused on embodied tasks by innovating pre-training approaches on human manipulation video datasets (Goyal et al., 2017; Grauman et al., 2022).”
InfluentialLIV: Language-Image Representations and Rewards for Robotic Control
Ma, Y., Liang, W., Som, V., Kumar, V., Zhang, A., Bastani, O., & Jayaraman, D. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2306.00958 · Cited by 312
“Our work is related to the literature on pre-training representations for control (Shah & Kumar, 2021; Cui et al., 2022; Parisi et al., 2022; Nair et al., 2022b; Xiao et al., 2022; Ma et al., 2022b; Fan et al., 2022; Majumdar et al., 2023).”
MethodSelf-citationBharadhwaj, H., Vakil, J., Sharma, M. K., Gupta, A. K., Tulsiani, S., & Kumar, V. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611293 · Cited by 88 · see citing works
“Most of these works focus on single-task settings [35], [36], [39], [40], or simulated robot environments [37], [40].”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15818 · Cited by 276 · see citing works
“Most works focus on pre-trained visual representations that can be used to initialize the encoder of the robot’s camera observations, either via supervised ImageNet classification [53], data augmentation [54, 55, 56, 57] or objectives that are tailored towards robotic control [58, 59, 60, 61, 62].”
BackgroundGen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot ManipulationPreprint
Bharadhwaj, H., Dwibedi, D., Gupta, A., Tulsiani, S., Doersch, C., Xiao, T., Shah, D., Xia, F., Sadigh, D., & Kirmani, S. (2024) · arXiv.org · doi:10.48550/arxiv.2409.16283 · Cited by 223 · see citing works
“To enable generalization, many recent works propose using pre-trained visual representations trained primarily on non-robot datasets [31, 32], for learning manipulation policies [5, 6, 6–9 , 33– 36 ].”
Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers
Wang, L., Chen, X., Zhao, J., & He, K.-M. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2409.20537 · Cited by 220
“Previous works have made significant progress in pre-training only the vision part of the policy on human videos [48, 53, 34, 61] and pre-training the full policy [6, 14, 55] with a unified model and dataset format (e.g. using languages [5]).”
BackgroundMethodResultInfluentialO’Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., et al. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10611477 · Cited by 210 · see citing works
Self-citationFlow as the Cross-Domain Manipulation InterfacePreprint
Xu, M., Xu, Z.-J., Xu, Y., Chi, C., Wetzstein, G., Veloso, M., & Song, S. (2024) · Conference on Robot Learning · doi:10.48550/arxiv.2407.15208 · Cited by 210 · see citing works
“A large body of work [10, 23, 24, 25] has studied leveraging cross-embodiment data to learn robotic policies.”
BackgroundTrack2Act: Predicting Point Tracks from Internet Videos Enables Generalizable Robot Manipulation
Bharadhwaj, H., Mottaghi, R., Gupta, A., & Tulsiani, S. (2024) · European Conference on Computer Vision · doi:10.1007/978-3-031-73116-7_18 · Cited by 200 · see citing works
“One common way of using data beyond robot interactions for efficient learning is to pre-train the visual representations which serve as backbones for the policy models [62, 63, 64, 65, 66] with passive human videos [23, 67] and image data [68].”
Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal GoalsPreprint
Reuss, M., Yagmurlu, Ö. E., Wenzel, F., & Lioutikov, R. (2024) · Robotics: Science and Systems Conference · doi:10.48550/arxiv.2407.05996 · Cited by 194 · see citing works
“A key area is learning universal vision representations or world-models, typically trained on large, diverse offline datasets using either masking strategies [22, 17, 12, 67, 50, 35] or contrastive objectives [14, 41, 52, 68, 40, 25, 2, 34, 46].”
BackgroundEgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos
Yang, R., Yu, Q.-X., Wu, Y.-C., Yan, R., Li, B.-R., Cheng, A.-C., Zou, X.-Y., Fang, Y.-H., Cheng, X.-X., Qiu, R.-Z., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2507.12440 · Cited by 192
“Motivated by egocentric vision, recent studies [60, 61, 62, 63, 64, 65, 66] pretrain representations using human videos, showing positive transfer.”
CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models
Huang, H., Lin, F., Hu, Y., Wang, S., & Gao, Y. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801352 · Cited by 66 · see citing works
“Many works employ vision models, pre-trained on large-scale image datasets, to generate visual representations for visuomotor control tasks [21], [31]–[34].”
On Bringing Robots HomePreprint
Shafiullah, N. M. (., Rai, A., Etukuru, H., Liu, Y.-Q., Misra, I., Chintala, S., & Pinto, L. (2023) · arXiv.org · doi:10.48550/arxiv.2311.16098 · Cited by 179 · see citing works
“We compare the effects of using our own visual encoder vs. a pretrained visual encoder trained on different datasets and algorithms, such as R3M [8], VC1 [9], and MVP [7], or even only pretraining on ImageNet-1K [59], in Section 3.4.1.”
BackgroundMethodInfluentialToward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis
Hu, Y.-F., Xie, Q.-T., Jain, V., Francis, J., Patrikar, J., Keetha, N., Kim, S., Xie, Y.-Q., Zhang, T.-Y., Zhao, S.-B., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2312.08782 · Cited by 158
“VC-1 [187] did a comprehensive study on the effectiveness of vision-based pre-training on policy learning.”
BackgroundSelf-citationGeneral Flow as Foundation Affordance for Scalable Robot Learning
Yuan, C., Wen, C., Zhang, T., & Gao, Y. (2024) · Conference on Robot Learning · doi:10.48550/arxiv.2401.11439 · Cited by 141
“Prior works [65, 56, 57, 90] employ large-scale visual pre-training to develop embodied-aware pretrained representations, but these demonstrate limited generalization [58, 38, 42].”
MethodReal-world robot applications of foundation models: a review
Kawaharazuka, K., Matsushima, T., Gambardella, A., Guo, J.-X., Paxton, C., & Zeng, A. (2024) · Adv. Robotics · doi:10.1080/01691864.2024.2408593 · Cited by 139 · see citing works
“Representative examples include R3M [3] and VC-1 [53].”
Octo: An Open-Source Generalist Robot Policy
Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., Luo, J., et al. (2024) · Robotics: Science and Systems XX · doi:10.15607/rss.2024.xx.090 · Cited by 129 · see citing works
SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Fang, H., Grotz, M., Pumacay, W., Wang, Y.-R., Fox, D., Krishna, R., & Duan, J. (2025) · International Conference on Machine Learning · doi:10.48550/arxiv.2501.18564 · Cited by 124
“Visual representations are integrated into robot learning through pre-training [17, 18, 19], co-training [20, 21, 22, 23], or frozen encoders [24, 25, 26], all of which effectively support policy training.”
Cheang, C., Chen, S., Cui, Z.-R., Hu, Y.-D., Huang, L.-Q., Kong, T., Li, H., Li, Y., Liu, Y., Ma, X., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2507.15493 · Cited by 120
“A popular approach is to initialize policies from pre-trained vision encoders [33, 50, 53, 61, 74] or more recently, pre-trained vision-language models [7, 9, 11, 29].”
Prediction with Action: Visual Policy Learning via Joint Denoising Process
Guo, Y., Hu, Y.-C., Zhang, J.-K., Wang, Y.-J., Chen, X.-Y., Lu, C., & Chen, J.-Y. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2411.18179 · Cited by 109
“Beyond directly acting as policy, pre-trained models can also guide policy learning from multiple aspects, such as providing good representations [45, 46, 47], providing reward signals [48, 49, 50], synthesizing goal images [18, 51], and predicting future sequences [17].”
Unifying 3D Vision-Language Understanding via Promptable Queries
Zhu, Z.-Y., Zhang, Z., Ma, X., Niu, X.-S., Chen, Y., Jia, B., Deng, Z.-D., Huang, S.-Y., & Li, Q. (2024) · European Conference on Computer Vision · doi:10.48550/arxiv.2405.11442 · Cited by 107
“To further demonstrate the capability of PQ3D , we also transfer it to an embodied agent for object navigation using the ObjNav task from CortexBench [50] and instruction-tune it with a large language model (LLM) Vicuna-7B using the instruction-following dataset from [32].”
BackgroundMethodVIMA: General Robot Manipulation with Multimodal Prompts
Jiang, Y., Gupta, A., Zhang, Z., Wang, G., Dou, Y., Chen, Y., Fei-Fei, L., Anandkumar, A., Zhu, Y., & Fan, L. (2022) · arXiv · doi:10.48550/arxiv.2210.03094 · Cited by 95 · see citing works
ICRT: In-Context Imitation Learning via Next-Token Prediction
Fu, M., Huang, H., Datta, G., Chen, L. Y., Panitch, W., Liu, F.-C., Li, H., & Goldberg, K. (2024) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra55743.2025.11128272 · Cited by 95 · see citing works
“However, most visual pre-trained networks are trained on ImageNet or human videos [24, 27, 52, 53], which exhibit a significant domain gap when compared to typical images from robot datasets, where the images frequently include robots or grippers.”
Theia: Distilling Diverse Vision Foundation Models for Robot Learning
Shang, J., Schmeckpeper, K., May, B. B., Minniti, M., Kelestemur, T., Watkins, D., & Herlant, L. (2024) · Conference on Robot Learning · doi:10.48550/arxiv.2407.20179 · Cited by 93
“We comprehensively evaluate Theia and baseline pre-trained models on the MuJoCo subset of Cor-texBench [20] to provide an overall assessment of pre-trained visual representations.”
BackgroundMethodResultInfluentialPoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators
Zeng, K.-H., Zhang, Z., Ehsani, K., Hendrix, R., Salvador, J., Herrasti, A., Girshick, R., Kembhavi, A., & Weihs, L. (2024) · Conference on Robot Learning · doi:10.48550/arxiv.2406.20083 · Cited by 91
“The prevalent approaches to building capable navigation agents include both end-to-end training [1, 6] and modular methods that leverage mapping [36, 37, 38] or off-the-shelf foundation models [39, 40, 41, 42, 43].”
BackgroundRISE: 3D Perception Makes Real-World Robot Imitation Simple and Effective
Wang, C., Fang, H., Fang, H.-S., & Lu, C. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801678 · Cited by 38 · see citing works
“One intuitive approach is to utilize pre-trained representation models for images [24, 25, 26, 31, 38] to convert them into 1D representations and map these transformed observations to the action space either through a BC policy [55] or non-parametric nearest neighbour [34].”
Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation
Bu, Q., Zeng, J., Chen, L., Yang, Y.-C., Zhou, G.-Y., Yan, J., Luo, P., Cui, H.-M., Ma, Y., & Li, H.-Y. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2409.09016 · Cited by 87
“We leverage the off-the-shelf pre-trained visual encoders ( i.e. , VC-1 [56], CLIP [23], and DINO [66]) to imbue the training process with visual priors.”
BackgroundMethodInfluentialSelf-citationQUAR-VLA: Vision-Language-Action Model for Quadruped Robots
Ding, P., Zhao, H., Zhang, W.-J., Song, W.-X., Zhang, M., Huang, S., Yang, N.-X., & Wang, D.-L. (2023) · European Conference on Computer Vision · doi:10.48550/arxiv.2312.14457 · Cited by 85
“The perception data is provided by a RealSense d435 camera installed in the front of the robot.”
BackgroundMethodIGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AI
Chen, X.-Y., Guo, J., He, T.-Y., Zhang, C., Zhang, P., Yang, D., Zhao, L., & Bian, J. (2024) · doi:10.48550/arxiv.2411.00785 · Cited by 84
“…on training representations for images/videos in self-supervised learning manner (He et al., 2021; Xiao et al., 2022; Radosavovic et al., 2022; Majumdar et al., 2023; Radford et al., 2021; Nair et al., 2022; Ma et al., 2023; Oquab et al., 2023; Darcet et al., 2023; Kirillov et al., 2023;…”
BackgroundEmbodied Understanding of Driving Scenarios
Zhou, Y.-S., Huang, L., Bu, Q., Zeng, J., Li, T., Qiu, H., Zhu, H.-Z., Guo, M.-Y., Qiao, Y., & Li, H.-Y. (2024) · European Conference on Computer Vision · doi:10.48550/arxiv.2403.04593 · Cited by 84
“BLIP2 [48] offers a universal and efficient pre-training strategy, guiding visual language pre-training from ready-made image encoders and frozen language models.”
BackgroundTrack2Act: Predicting Point Tracks from Internet Videos enables Diverse Zero-shot Robot ManipulationPreprint
Bharadhwaj, H., Mottaghi, R., Gupta, A., & Tulsiani, S. (2024) · arXiv.org · doi:10.48550/arxiv.2405.01527 · Cited by 83 · see citing works
“One common way of using data beyond robot interactions for efficient learning is to pre-train the visual representations which serve as backbones for the policy models [62, 63, 64, 65, 66] with passive human videos [23, 67] and image data [68].”
RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning
Lei, K., Li, H.-Y., Yu, D., Wei, Z.-Y., Guo, L., Jiang, Z.-N., Wang, Z., Liang, S., & Xu, H.-Z. (2025) · Sci. Robotics · doi:10.48550/arxiv.2510.14830 · Cited by 82
“Given a schedule { α t } Tt =1 with α t = 1 − β t , a sample x 0 drawn from the clean distribution, the forward noising process follows a closed form: A denoiser ε θ is trained to recognize the noise inside the noisy sample via (3) to recover the clean sample.”
Generating meaning: active inference and the scope and limits of passive AI
Pezzulo, G., Parr, T., Cisek, P., Clark, A., & Friston, K. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.10.002 · Cited by 80 · see citing works
Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Plans
Bharadhwaj, H., Gupta, A., Kumar, V., & Tulsiani, S. (2023) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra57147.2024.10610288 · Cited by 80 · see citing works
“Some prior robot learning approaches have also investigated leveraging out-of-domain (human) data, primarily for learning visual representations [2, 3, 4] and robotic affordances [5, 6, 7, 8].”
BackgroundEhsani, K., Gupta, T., Hendrix, R., Salvador, J., Weihs, L., Zeng, K.-H., Singh, K. P., Kim, Y., Han, W., Herrasti, A., et al. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52733.2024.01537 · Cited by 76
“[45] have comprehensively studied the impact of visual backbones on embodied performance.”
BackgroundPoint Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning
Zhu, H., Wang, Y.-T., Huang, D., Ye, W., Ouyang, W.-L., & He, T. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2402.02500 · Cited by 74
“In addition, we adopt the evaluation methodology from VC-1 (Majumdar et al., 2023) encompassing two key metrics: Mean Success and Mean Rank .”
BackgroundMethodInfluentialMasquerade: Learning from In-the-wild Human Videos using Data-Editing
Lepert, M., Fang, J.-Y., & Bohg, J. (2025) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra57385.2026.11696751 · Cited by 73
“Masked auto-encoding approaches like MVP [3] and VC-1 [4] adapt MAE transformers [15] to human clips.”
RoboHive: A Unified Framework for Robot LearningPreprint
Kumar, V., Shah, R. M., Zhou, G.-Y., Moens, V., Caggiano, V., Vakil, J., Gupta, A., & Rajeswaran, A. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2310.06828 · Cited by 73 · see citing works
Self-citationDynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control
Cui, Z. J., Pan, H.-K., Iyer, A., Haldar, S., & Pinto, L. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2409.12192 · Cited by 69
“These methods have been extended to decision-making through effective downstream policy learning [7–11 , 6].”
InfluentialHand-Object Interaction Pretraining from Videos
Singh, H. G., Loquercio, A., Sferrazza, C., Wu, J., Qi, H., Abbeel, P., & Malik, J. (2024) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra55743.2025.11127811 · Cited by 69 · see citing works
“Inspired by successes in computer vision [1] and natural language processing [70], the robot learning community has recently focused on pretraining representations on large video datasets like Ego4D [49] and fine-tuning these representations on downstream tasks [4–6 , 8].”
Self-citationTAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Models
Liu, Z., Zhang, J., Asadi, K., Liu, Y., Zhao, D., Sabach, S., & Fakoor, R. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.05905 · Cited by 69
“Apart from the CLIP-ViT pretrained encodings as we adopted in our main results, two other initialization of weights were considered: one sourced from the Visual Cortex 1 (VC-1) (Majumdar et al., 2023b), recognized for being a leading pretrained C.4 R ANK S IZE A BLATION S TUDY In order to understand the impact of rank…”
Pure Vision Language Action (VLA) Models: A Comprehensive Survey
Zhang, D.-P., Sun, J., Hu, C., Wu, X., Yuan, Z., Zhou, R., Shen, F., & Zhou, Q.-G. (2025) · arXiv.org · doi:10.48550/arxiv.2509.19012 · Cited by 67
“VC-1 [223] 2024 Studies pre-training data scale using 4,000+ hours of video with MAE-trained transformers.”
Efficient Data Collection for Robotic Manipulation via Compositional Generalization
Gao, J., Xie, A., Xiao, T., Finn, C., & Sadigh, D. (2024) · Robotics: Science and Systems Conference · doi:10.48550/arxiv.2403.05110 · Cited by 64
“Much prior work has studied improving generalization in robot learning, such as by using pre-trained visual representations [36, 33, 30, 39, 11, 31, 20], or pre-training on diverse robot data [35, 34].”
Training Strategies for Efficient Embodied ReasoningPreprint
Chen, W., Belkhale, S., Mirchandani, S., Mees, O., Driess, D., Pertsch, K., & Levine, S. (2025) · arXiv.org · doi:10.48550/arxiv.2505.08243 · Cited by 63 · see citing works
“Numerous authors have suggested pre-training base models on egocentric/embodied data [57, 58] or grounded spatial reasoning tasks [59, 60], as the resulting representations might be more conducive to learning control.”
An Unbiased Look at Datasets for Visuo-Motor Pre-Training
Dasari, S., Srirama, M. K., Jain, U., & Gupta, A. (2023) · Conference on Robot Learning · doi:10.48550/arxiv.2310.09289 · Cited by 63
“We also evaluate pre-trained weights downloaded from two SOTA baselines (VC-1 [15], MVP [22]).”
BackgroundMethodInfluentialScene-Driven Multimodal Knowledge Graph Construction for Embodied AI
Song, Y., Sun, P., Liu, H., Li, Z., Song, W., Xiao, Y., & Zhou, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3399746 · Cited by 53 · see citing works
“In contrast, robot equipped with CKR-ConceptNet starts an incorrect trajectories at step 0 and stops at an incorrect point at step n .”
MethodViTaMIn: Learning Contact-Rich Tasks Through Robot-Free Visuo-Tactile Manipulation Interface
Liu, F.-C., Li, C., Qin, Y., Shaw, A., Xu, J., Abbeel, P., & Chen, R. (2025) · arXiv.org · doi:10.48550/arxiv.2504.06156 · Cited by 61
Self-citationJia, Y.-R., Liu, J.-M., Chen, S.-X., Gu, C.-Y., Wang, Z., Luo, L., Lee, L., Wang, P.-W., Wang, Z.-Y., Zhang, R.-R., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2411.18623 · Cited by 61
“We compare various baseline methods, such as robotic 2D representation meth-ods [52, 54, 61], 3D representation methods [58, 59, 96], and 3D imitation learning policies [23, 24, 36].”
InfluentialEmergence of Human to Robot Transfer in Vision-Language-Action Models
Kareer, S., Pertsch, K., Darpinian, J., Hoffman, J., Xu, D.-F., Levine, S., Finn, C., & Nair, S. (2025) · Robotics · doi:10.48550/arxiv.2512.22414 · Cited by 60
From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Control
Shentu, Y., Wu, P., Rajeswaran, A., & Abbeel, P. (2024) · IEEE/RJS International Conference on Intelligent RObots and Systems · doi:10.1109/iros58592.2024.10801683 · Cited by 59 · see citing works
“We are also starting to see large models and representations custom trained for robotics [1], [28], [29], [30].”
Self-citationExploring Embodied Intelligence in Soft Robotics: A Review
Zhao, Z., Wu, Q.-X., Wang, J., Zhang, B.-T., Zhong, C.-L., & Zhilenkov, A. (2024) · Biomimetics · doi:10.3390/biomimetics9040248 · Cited by 58 · see citing works
What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?
Burns, K., Witzel, Z., Hamid, J. I., Yu, T.-H., Finn, C., & Hausman, K. (2023) · Conference on Robot Learning · doi:10.48550/arxiv.2312.12444 · Cited by 58
“Majumdar et al. (2023) benchmarks the performance of pre-trained visual representations on a handful of manipulation environments, but they focus on in-domain performance and also investigate navigation environments.”
BackgroundHRP: Human Affordances for Robotic Pre-TrainingPreprint
Srirama, M. K., Dasari, S., Bahl, S., & Gupta, A. (2024) · Robotics: Science and Systems Conference · doi:10.48550/arxiv.2407.18911 · Cited by 57 · see citing works
“One potential issue is that roboticists often use the same exact pre-training methods from the vision community, but merely apply them to a different data mix (e.g., VC-1 [56] applies MAE [39] to Ego4D [32]).”
BackgroundMethodInfluentialTransfer learning in robotics: An upcoming breakthrough? A review of promises and challenges
Jaquier, N., Welle, M. C., Gams, A., Yao, K.-P., Fichera, B., Billard, A., Ude, A., Asfour, T., & Kragic, D. (2023) · Int. J. Robotics Res. · doi:10.1177/02783649241273565 · Cited by 57 · see citing works
“The increased performance of RT-2 compared to RT-1 and other adjusted baseline models (such as VC-1 (Majumdar et al. 2023), R3M (Nair et al. 2022), MOO (Stone et al. 2023)) is attributed to the vision-language backbone combining co-finetuning the pre-trained model jointly on robotics and web data, so that the model con…”
GR-MG: Leveraging Partially-Annotated Data via Multi-Modal Goal-Conditioned Policy
Li, P.-Y., Wu, H.-T., Huang, Y., Cheang, C., Wang, L., & Kong, T. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3526436 · Cited by 56 · see citing works
Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning
Wu, J., Ma, H., Deng, C., & Long, M.-S. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.18499 · Cited by 56
“Recent work has explored general-2 purpose visual pre-training using in-the-wild datasets [69, 36, 77, 53, 86, 49, 45] to accelerate policy learning in model-free manners.”
BackgroundSUGAR : Pre-training 3D Visual Representations for Robotics
Chen, S., García, R., Laptev, I., & Schmid, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01709 · Cited by 15 · see citing works
RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization
Yuan, Z.-C., Yang, S., Hua, P., Chang, C., Hu, K., Wang, X.-L., & Xu, H. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2307.10224 · Cited by 52
PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic ManipulationPreprint
Liu, Y.-Z., Zhu, J.-Y., Mo, Y.-C., Li, G., Cao, X., Jin, J., Shen, Y.-F., Li, Z.-Y., Yu, T., Yuan, W., et al. (2026) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.2601.07060 · Cited by 51 · see citing works
“Recent works have shifted towards learning continuous progress representations directly from large-scale video data [65, 66, 71].”
AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost ExoskeletonsPreprint
Fang, H., Wang, C., Wang, Y., Chen, J., Xia, S., Lv, J., He, Z., Yi, X., Guo, Y., Zhan, X., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.03081 · Cited by 2 · see citing works
“Some approaches [2, 62, 63, 64, 68, 77, 87, 111] focus on visual representation learning from human videos, while others [4, 10, 29, 75, 103, 108] propose pre-training the policy backbone with auxiliary video or latent prediction objectives.”
Industrial Foundation Models (IFMs) for intelligent manufacturing: A systematic review
Zhao, S., Liu, S., Jiang, Y., Zhao, B. M., Lv, Y., Zhang, J., Wang, L., & Zhong, R. Y. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.06.011 · Cited by 46 · see citing works
What foundation models can bring for robot learning in manipulation: A survey
Li, D., Jin, Y., Sun, Y.-H., Yong, A., Yu, H., Shi, J., Hao, X., Peng, H., Liu, H., Li, X., et al. (2025) · The International Journal of Robotics Research · doi:10.1177/02783649251390579 · Cited by 14 · see citing works
“To compare different pre-trained visual representations, benchmarks are established by CORTEXBENCH (Majumdar et al. (2023)) and EmbCLIP (Khandelwal et al. (2022)) to assess which model could provide a better ”artificial visual cortex” for manipulation tasks.”
Lossless Adaptation of Pretrained Vision Models For Robotic Manipulation
Sharma, M., Fantacci, C., Zhou, Y.-X., Koppula, S., Heess, N., Scholz, J., & Aytar, Y. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2304.06600 · Cited by 48
“Out of Domain Representation Learning for Control: An alternative set of works have emphasized the use of in-the-wild visual datasets for representation learning (Parisi et al., 2022; Shridhar et al., 2022; Khandelwal et al., 2022; Yen-Chen et al., 2020; Majumdar et al., 2023).”
BackgroundViNT: A Foundation Model for Visual Navigation
Shah, D., Sridhar, A., Dashora, N., Stachowicz, K., Black, K., Hirose, N., & Levine, S. (2023) · arXiv · doi:10.48550/arxiv.2306.14846 · Cited by 45 · see citing works
Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videos
Luo, R. M., Xue, Z., Dimakis, A., & Grauman, K. (2024) · European Conference on Computer Vision · doi:10.48550/arxiv.2403.06351 · Cited by 45
“Indeed, recent work demonstrates that the human ego view is valuable for robot learning Bharadhwaj et al. (2023); Bahl et al. (2022); Majumdar et al. (2023); Nair et al. (2022); Mandikal and Grauman (2021).”
BackgroundSPA: 3D Spatial-Awareness Enables Effective Embodied Representation
Zhu, H., Yang, H.-H., Wang, Y.-T., Yang, J.-G., Wang, L.-M., & He, T. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.08208 · Cited by 44
“Inspired by computer vision, recent work has applied methods such as contrastive learning (Nair et al., 2022; Yang et al., 2023) and masked autoen-coders (Radosavovic et al., 2023; Majumdar et al., 2023; Karamcheti et al., 2023; Yang et al., 2024b) to embodied AI.”
MethodResultInfluentialUniversal Visual Decomposer: Long-Horizon Manipulation Made Easy
Zhang, Z., Li, Y., Bastani, O., Gupta, A., Jayaraman, D., Ma, Y. J., & Weihs, L. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611125 · Cited by 10 · see citing works
“…require strong assumptions about tasks, datasets, or robotic platforms [3,10,11,16,17,26,29,34,39– 41 , In order to decompose any long-horizon task using vision, general knowledge about visual task progression that can discern embedded subtasks in long, unsegmented task videos must be acquired.”
BackgroundSelf-citationUAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation
Tang, Y., Huang, W.-L., Wang, Y.-K., Li, C., Yuan, R., Zhang, R.-H., Wu, J.-J., & Li, F.-F. (2025) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra55743.2025.11128868 · Cited by 43 · see citing works
Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets
Jiang, G., Sun, Y., Huang, T., Li, H.-Y., Liang, Y.-Y., & Xu, H. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.22325 · Cited by 43
“Methods like VC-1 and MVP have prolonged training times due to their use of MAE (Feichtenhofer et al., 2022).”
BackgroundMethodInfluentialVisual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in SpacesPreprint
Luo, G., Yang, G.-L., Gong, Z.-Y., Chen, G.-Z., Duan, H., Cui, E.-F., Tong, R., Hou, Z., Zhang, T.-Y., Chen, Z., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2506.00123 · Cited by 42 · see citing works
“Some approaches [78, 85, 71, 89, 59] focus on the enhancement of individual components, such as pretrained visual representations, dynamics learning, and reasoning.”
Wang, H., Tan, A. H., & Nejat, G. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3412638 · Cited by 40 · see citing works
“The robot planned a global path using A ∗ and followed this path using DWA [30] as the local planner.”
MethodHuman2Robot: Learning Robot Actions from Paired Human-Robot Videos
Xie, S.-C., Cao, H.-D., Weng, Z.-J., Xing, Z., Shen, S., Leng, J.-Q., Qiu, X., Fu, Y.-W., Wu, Z.-X., & Jiang, Y.-G. (2025) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2502.16587 · Cited by 39
“Researchers propose to learn representations from human videos to assist in task execution [36, 60, 65].”
Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training
He, H., Bai, C., Pan, L., Zhang, W.-N., Zhao, B., & Li, X.-L. (2024) · Neural Information Processing Systems · doi:10.52202/079017-0981 · Cited by 39
“…and stands as the state-of-the-art (SOTA) visual representation specifically designed for manipulation tasks; (ii) VC-1-Diffusion utilizes VC-1 [58] encoder (ViT-L) to extract image representations, which is also trained on large-scale egocentric videos [27] and ImageNet [71] using Masked…”
MethodAirExo: Low-Cost Exoskeletons for Learning Whole-Arm Manipulation in the Wild
Fang, H., Fang, H.-S., Wang, Y., Ren, J., Chen, J., Zhang, R., Wang, W., & Lu, C. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10610799 · Cited by 38 · see citing works
“3) Methods: We employ VINN [26] and its variants that alter the visual representations [22, 23, 29] as non-parametric methods.”
MethodMitigating the Human-Robot Domain Discrepancy in Visual Pre-training for Robotic Manipulation
Zhou, J.-M., Ma, T.-L., Lin, K.-Y., Wang, Z.-F., Qiu, R.-H., & Liang, J.-W. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.02100 · Cited by 38 · see citing works
“In parallel, several studies [13, 24, 33, 36, 43–45, 49, 51, 67] have proposed methods to learn a generalizable representation from large-scale human datasets, such as Ego4D [21] and Kinetics [8].”
A Universal Semantic-Geometric Representation for Robotic Manipulation
Zhang, T., Hu, Y.-D., Cui, H.-C., Zhao, H., & Gao, Y. (2023) · Conference on Robot Learning · doi:10.48550/arxiv.2306.10474 · Cited by 38
Cage: Causal Attention Enables Data-Efficient Generalizable Robotic Manipulation
Xia, S., Fang, H.-J., Lu, C.-W., & Fang, H. (2024) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra55743.2025.11127739 · Cited by 35 · see citing works
“Alternatively, some literature [30, 31, 33, 41, 45] proposes decoupling perception and control in the policy by using pre-trained 2D visual representations for manipulations, but they offer limited advantages over VFMs in generalization evaluations [3].”
Semantically controllable augmentations for generalizable robot learning
Chen, Z., Zhao, M.-D., Bharadhwaj, H., Sharma, M., Song, S., Gupta, A., & Kumar, V. (2024) · Int. J. Robotics Res. · doi:10.1177/02783649241273686 · Cited by 35 · see citing works
“Many works focus on single-task settings (Nair et al. 2022a; Parisi et al. 2022; Sharma et al. 2023; Hansen et al. 2022), or simulated robot environments (Hansen et al. 2022; Majumdar et al. 2023).”
Feat2GS: Probing Visual Foundation Models with Gaussian Splatting
Chen, Y., Chen, X., Chen, A.-P., Pons-Moll, G., & Xiu, Y. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.00595 · Cited by 33
“There is a line of works that try to answer these questions through multi-view object consistency [7], spatial visual question answering [27, 122], visual perspective taking [54] and robot learning [57, 79, 119].”
HMD2: Environment-Aware Motion Generation from Single Egocentric Head-Mounted Device
Guzov, V., Jiang, Y., Hong, F.-Z., Pons-Moll, G., Newcombe, R. A., Liu, C. K., Ye, Y., & Ma, L.-N. (2024) · International Conference on 3D Vision · doi:10.1109/3dv66043.2025.00132 · Cited by 33 · see citing works
“To explain our choice of CLIP [56] feature as a feature encoder, we additionally trained two versions of our method with image features produced by DINOv2 [50] and VC-1 [47] feature encoders.”
Nayebi, A., Rajalingham, R., Jazayeri, M., & Yang, G. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.11772 · Cited by 33
On the Utility of Koopman Operator Theory in Learning Dexterous Manipulation Skills
Han, Y., Xie, M., Zhao, Y., & Ravichandar, H. (2023) · Conference on Robot Learning · doi:10.48550/arxiv.2303.13446 · Cited by 33
“, the benchmark tasks in [40]) to further understand the trade-offs between KODex and other imitation learning baselines.”
BackgroundEhsani, K., Gupta, T., Hendrix, R., Salvador, J., Weihs, L., Zeng, K.-H., Singh, K. P., Kim, Y., Han, W., Herrasti, A., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2312.02976 · Cited by 33 · see citing works
“[45] have comprehensively studied the impact of visual backbones on embodied performance.”
BackgroundTD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning
Bagatella, M., Pirotta, M., Touati, A., Lazaric, A., & Tirinzoni, A. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2510.00739 · Cited by 32
“These experiments thus recall the standard setting in which pre-trained visual representations are evaluated (Nair et al., 2022; Ma et al., 2023; Majumdar et al., 2023).”
Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulation
Mejia, J., Dean, V., Hellebrekers, T., & Gupta, A. (2024) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra57147.2024.10611305 · Cited by 32 · see citing works
“While recent work has leveraged pretraining on large-scale video datasets to create reusable vision representations for robot learning [1]–[3], there has been little focus on large-scale pretraining for other modalities such as tactile sensing.”
BackgroundJEPA-VLA: Video Predictive Embedding is Needed for VLA Models
Miao, S.-C., Feng, N.-Y., Wu, J., Lin, Y.-Y., He, X., Li, D., & Long, M.-S. (2026) · arXiv.org · doi:10.48550/arxiv.2602.11832 · Cited by 31
Self-citationH2R: A Human-to-Robot Data Augmentation for Robot Pre-training from Videos
Li, G.-R., Lyu, Y., Liu, Z.-Y., Hou, C., Zhang, J., & Zhang, S.-H. (2025) · arXiv.org · doi:10.48550/arxiv.2505.11920 · Cited by 31
“Notably, MAE observes a substantial 25.5% gain on the MoveCan task.”
InfluentialMoRE: Unlocking Scalability in Reinforcement Learning for Quadruped Vision-Language-Action Models
Zhao, H., Song, W.-X., Wang, D.-L., Tong, X., Ding, P., Cheng, X.-L., & Ge, Z.-Y. (2025) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra55743.2025.11128601 · Cited by 31 · see citing works
“We compared our method with three previous state-of-the-art approaches on the quadruped robot VLA task: QUART [24], VC-1 [36], and CLIP [35].”
Embodied AI Agents: Modeling the World
Fung, P., Bachrach, Y., Celikyilmaz, A., Chaudhuri, K., Chen, D., Chung, W., Dupoux, E., Gong, H., Jégou, H., Lazaric, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.22355 · Cited by 29 · see citing works
“For some capabilities, such as visual navigation, simulation benchmark performance has positively correlated with real-world metrics (Kadian et al., 2020; Silwal et al., 2024) especially when utilizing encoders pretrained on real data - such as VC-1 (Majumdar et al., 2023) - in both sim and real experiments.”
Self-citationReinforcement Learning with Foundation Priors: Let Embodied Agent Efficiently Learn on Its Own
Ye, W.-R., Zhang, Y., Weng, H.-Y., Gu, X., Wang, S.-J., Zhang, T., Wang, M.-C., Abbeel, P., & Gao, Y. (2023) · Conference on Robot Learning · doi:10.48550/arxiv.2310.02635 · Cited by 29
“Several works leverage pre-trained visual representations to initialize perception encoders or extract latent image states [43, 44, 45, 46], while others use pre-trained LLMs or VLMs for linguistic instruction encoding [47, 48, 49].”
Self-citationTransferring Foundation Models for Generalizable Robotic Manipulation
Yang, J., Tan, W., Jin, C., Yao, K., Liu, B., Fu, J., Song, R., Wu, G., & Wang, L. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00201 · Cited by 17 · see citing works
Zhao, B., Fang, J., Dai, Z., Wang, Z., Zha, J., Zhang, W., Gao, C., Wang, Y., Cui, J., Chen, X., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1558 · Cited by 28 · see citing works
“…al., 2024) that primarily focused on 2D images, static point cloud or language-based spatial understanding, human comprehension of the world is grounded in continuous visual perception (Zhang et al., 2024; Liao et al., 2024; Majumdar et al., 2023), akin to embodied cognition through video streams.”
MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos
Shah, R. M., Liu, S.-J., Wang, Q., Jiang, Z.-Y., Kumar, S., Seo, M., Martín-Martín, R., & Zhu, Y.-K. (2025) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra57385.2026.11696685 · Cited by 28
“Prior work has explored various strategies to extract knowledge from human videos, such as learning visual representations [36–38] and deriving reward functions [39–42].”
3D-MVP: 3D Multiview Pretraining for Manipulation
Qian, S., Mo, K., Blukis, V., Fouhey, D. F., Fox, D., & Goyal, A. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.02098 · Cited by 28
“Specifically, the robotics community has trended towards learning representations us-ing state-of-the-art self-supervised vision algorithms on diverse interaction datasets [8, 17, 39], and finetune the network on robotics tasks [9, 28–30, 33, 37, 42].”
Lift3D Policy: Lifting 2D Foundation Models for Robust 3D Robotic Manipulation
Jia, Y.-R., Liu, J.-M., Chen, S.-X., Gu, C.-Y., Wang, Z., Luo, L., Li, X., Wang, P.-W., Wang, Z.-Y., Zhang, R.-R., et al. (2025) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.01617 · Cited by 27 · see citing works
“Previous robotic MAE methods typically use low-level RGB information [46, 54, 56] as the reconstruction target.”
InfluentialSpatiotemporal Predictive Pre-training for Robotic Motor Control
Yang, J.-G., Liu, B., Fu, J., Pan, B., Wu, G.-S., & Wang, L.-M. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-026-02948-3 · Cited by 27
“Recently, inspired by the success of transfer learning in CV, many works [69, 73, 65, 58, 59, 19] have explored developing a pre-trained visual representation (PVR) using large-scale out-of-domain data for various robotic motor control tasks.”
Influential
- Authors
- Majumdar, A., Yadav, K., Arnaud, S., Ma, Y. J., Chen, C., Silwal, S., Jain, A., Berges, V.-P., Abbeel, P., Malik, J., Batra, D., Lin, Y., Maksymets, O., Rajeswaran, A., & Meier, F.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 655-677
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2303.18240
- OpenAlex
- W7133188016
- Cited by
- 49
- References
- 83
- Citations by year
- 2026: 12 · 2025: 14 · 2024: 21 · 2023: 2
- FWCI
- 25.21
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Keywords
- embodied AI, visual cortex, artificial intelligence, visual perception, neuromorphic computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
