The DOI matches its DataCite record; 68 of the 83 references it cites were found in a registry or an index.
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?
Open accessAbstract
We present the largest and most comprehensive empirical study of pre-trained visual representations (PVRs) or visual 'foundation models' for Embodied AI. First, we curate CortexBench, consisting of 17 different tasks spanning locomotion, navigation, dexterous, and mobile manipulation. Next, we systematically evaluate existing PVRs and find that none are universally dominant. To study the effect of pre-training data size and diversity, we combine over 4,000 hours of egocentric videos from 7 different sources (over 4.3M images) and ImageNet to train different-sized vision transformers using Masked Auto-Encoding (MAE) on slices of this data. Contrary to inferences from prior work, we find that scaling dataset size and diversity does not improve performance universally (but does so on average). Our largest model, named VC-1, outperforms all prior PVRs on average but does not universally dominate either. Next, we show that task- or domain-specific adaptation of VC-1 leads to substantial gains, with VC-1 (adapted) achieving competitive or superior performance than the best known results on all of the benchmarks in CortexBench. Finally, we present real-world hardware experiments, in which VC-1 and VC-1 (adapted) outperform the strongest pre-existing PVR. Overall, this paper presents no new techniques but a rigorous systematic evaluation, a broad set of findings about PVRs (that in some cases, refute those made in narrow domains in prior work), and open-sourced code and models (that required over 10,000 GPU-hours to train) for the benefit of the research community.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“| Octo (Ours) | 70% | 75% | 50% | 60% | 100% | 80% | 72% |”
From Octo: An Open-Source Generalist Robot Policy (2024) · cites this paper as [57]
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 7×
“We choose the number of epochs per run such that the number of model updates remain constant across all runs and match the number of model updates taken by MAE on the ImageNet dataset.”
MethodInfluentialAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 12×
“VC-1 is superior to the second-best model ( Ego4D+MN ViT-L, Table 4 row 9), which has an average rank of 3.1.”
MethodInfluentialMomentum Contrast for Unsupervised Visual Representation LearningPreprint
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. B. (2019) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1911.05722 · Cited by 16,250
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 7×
“3 However, curiously, the kinds of PVRs that are locally-dominant in CORTEXBENCH differ significantly in the size and type of pre-training datasets: CLIP (Radford et al., 2021) was pretrained on 400M image-text pairs from the web; MVP (Radosavovic et al.”
BackgroundInfluentialA simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · cited 1×
“These algorithms use contrastive [21, 22], distillation-based [19, 20], or reconstructive [18, 23] objectives for training.”
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · cited 10×
“Since VC-1 was trained with MAE (He et al., 2021), it captures features that are generally useful for reconstructing images.”
BackgroundMethodInfluentialFukushima, K. (1980) · Biological Cybernetics · doi:10.1007/bf00344251 · Cited by 5,759 · cited 2×
“In this work, we ask the same question that Fukushima [1, 2] asked nearly 50 years ago – how do we design an artificial visual cortex , the module in a computational system that (together with a policy) enables an agent to convert camera input into actions?”
BackgroundEmerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 1×
“End-to-end (E2E) fine-tuning with a task-specific loss function can in-principle capture both of the aforementioned benefits of adaptation, and is widely used in computer vision literature (He et al., 2020; Caron et al., 2021; He et al., 2021; Baevski et al., 2022b).”
MethodChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“We use a broad definition of adaptation [55], which, in the context of large pre-trained foundation models, can take several forms from simple prompting [56], to selectively updating some or all weights of the backbone [5, 35, 57].”
MethodOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., & Joulin, A. (2020) · arXiv · doi:10.48550/arxiv.2006.09882 · Cited by 2,499 · cited 2×
“The last few years have seen increasing interest in the self-supervised learning (SSL) of visual representations (He et al., 2021; Caron et al., 2020; Baevski et al., 2022b; Chen et al., 2020; 2021).”
BackgroundAn Empirical Study of Training Self-Supervised Vision Transformers
Chen, X., Xie, S., & He, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00950 · Cited by 1,648
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., & Farhadi, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989381 · Cited by 1,533 · cited 1×
“We consider two semantic navigation tasks in Habitat: image-goal navigation (ImageNav) (Zhu et al., 2017) and object-goal navigation (ObjectNav) (Batra et al.”
MethodLearning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J. D., Todorov, E., & Levine, S. (2017) · Robotics: Science and Systems Conference · doi:10.48550/arxiv.1709.10087 · Cited by 1,522 · cited 2×
“, 2020), dexterous manipulation (Rajeswaran et al., 2018), multifinger coordinated manipulation (Wüthrich et al.”
BackgroundSelf-citationThe “Something Something” Video Database for Learning and Evaluating Visual Common Sense
Goyal, R., Kahou, S. E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.622 · Cited by 1,471 · cited 2×
“One cluster of datasets – Ego4D [49], 100 Days of Hands (100DOH) [50], Something-Something v2 (SS-V2) [51], and Epic Kitchens [52] – contains videos of people manipulating objects and is comparable to the datasets used in MVP [8].”
MethodBEiT: BERT Pre-Training of Image Transformers
Bao, H., Dong, L., Piao, S., & Wei, F. (2021) · arXiv · doi:10.48550/arxiv.2106.08254 · Cited by 1,392
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and LanguagePreprint
Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., & Auli, M. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2202.03555 · Cited by 1,303 · cited 2×
“End-to-end (E2E) fine-tuning with a task-specific loss function can in-principle capture both of the aforementioned benefits of adaptation, and is widely used in computer vision literature [18, 20, 58, 59].”
MethodHabitat: A Platform for Embodied AI Research
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00943 · Cited by 1,237 · cited 2×
“The dataset is comprised of 80 training, 20 validation, and 20 testing scenes and utilizes the Habitat simulator [15, 65] and HM3D [68] environments.”
MethodSelf-citationFine-Tuning can Distort Pretrained Features and Underperform Out-of-DistributionPreprint
Kumar, A., Raghunathan, A., Jones, R., Ma, T.-Y., & Liang, P. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2202.10054 · Cited by 1,110 · cited 1×
“, 2022), to selectively updating some or all weights of the backbone (Kumar et al., 2022; Hansen et al., 2022a; Yadav et al., 2023).”
BackgroundGibson Env: Real-World Perception for Embodied AgentsPreprint
Xia, F., Zamir, A., He, Z.-Y., Sax, A., Malik, J., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1808.10654 · Cited by 1,076
Self-citationZhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891 · cited 1×
“Several works have showed that scaling model and dataset size improves performance on vision tasks like image classification (Zhai et al., 2022; Tian et al., 2021; Goyal et al., 2021).”
BackgroundCognitron: A self-organizing multilayered neural network
Fukushima, K. (1975) · Biological cybernetics · doi:10.1007/bf00342633 · Cited by 639 · cited 2×
“In this work, we ask the same question that Fukushima [1, 2] asked nearly 50 years ago – how do we design an artificial visual cortex , the module in a computational system that (together with a policy) enables an agent to convert camera input into actions?”
BackgroundEgo4D: Around the World in 3,000 Hours of Egocentric Video
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01842 · Cited by 630 · cited 1×
“These datasets include Ego4D [22], 100 Days of Hands (100DOH) [23], Something-Something v2 (SS-V2) [24], and Epic Kitchens [25].”
MethodSelf-citationTassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. D. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1801.00690 · Cited by 497 · cited 2×
“We curate C ORTEX B ENCH , a benchmark for evaluating PVRs, consisting of 17 tasks spanning low-level locomotion [10], table-top manipulation [11], dexterous manipulation [12], multi-finger coordinated manipulation [13], indoor visual navigation [14], and mobile manipulation [15].”
BackgroundMethodR3M: A Universal Visual Representation for Robot Manipulation
Nair, S., Rajeswaran, A., Kumar, V., Finn, C., & Gupta, A. (2022) · arXiv · doi:10.48550/arxiv.2203.12601 · Cited by 492 · cited 14×
“WefindthatR3Menablesdataefficientimitationlearningacrossseveralsimulatedandreal-worldrobotmanipulationtasks.”
MethodInfluentialSelf-citationMeta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
Yu, T., Quillen, D., He, Z., Julian, R., Narayan, A., Shively, H., Bellathur, A., Hausman, K., Finn, C., & Levine, S. (2019) · arXiv · doi:10.48550/arxiv.1910.10897 · Cited by 450 · cited 2×
“MetaWorld (MW) (Yu et al., 2020) is a collection of tasks in which an agent commands a Sawyer robot arm to manipulate objects in a tabletop environment.”
BackgroundUnderstanding Human Hands in Contact at Internet ScalePreprint
Shan, D., Geng, J., Shu, M., & Fouhey, D. F. (2020) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.2006.06669 · Cited by 433 · cited 1×
“, 2022), 100 Days of Hands (100DOH) (Shan et al., 2020), Something-Something v2 (SS-V2) (Goyal et al.”
Real-World Robot Learning with Masked Visual Pre-trainingPreprint
Radosavovic, I., Xiao, T.-T., James, S., Abbeel, P., Malik, J., & Darrell, T. (2022) · Conference on Robot Learning · doi:10.48550/arxiv.2210.03109 · Cited by 389 · cited 5×
“, 2021) was pretrained on 400M image-text pairs from the web; MVP (Radosavovic et al., 2022) on 4.”
MethodInfluentialSelf-citationDD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion FramesPreprint
Wijmans, E., Kadian, A., Morcos, A. S., Lee, S., Essa, I. A., Parikh, D., Savva, M., & Batra, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00357 · Cited by 358
Self-citationLIV: Language-Image Representations and Rewards for Robotic Control
Ma, Y., Liang, W., Som, V., Kumar, V., Zhang, A., Bastani, O., & Jayaraman, D. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2306.00958 · Cited by 312 · cited 2×
“[28] focuses on self-supervised representation learning for goal-conditioned value functions using language-aligned videos.”
BackgroundSelf-citationVIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-TrainingPreprint
Ma, Y. J., Sodhani, S., Jayaraman, D., Bastani, O., Kumar, V., & Zhang, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.00030 · Cited by 275 · cited 3×
“Inspired by the advancements in self-supervised learning, recent work has incorporated visual representation learning into the training pipelines for EAI agents [3–9].”
BackgroundMethodInfluentialSelf-citationScaling Egocentric Vision: The EPIC-KITCHENS Dataset
Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2018) · arXiv · doi:10.48550/arxiv.1804.02748 · Cited by 274 · cited 1×
“One cluster of datasets – Ego4D [49], 100 Days of Hands (100DOH) [50], Something-Something v2 (SS-V2) [51], and Epic Kitchens [52] – contains videos of people manipulating objects and is comparable to the datasets used in MVP [8].”
MethodThe Unsurprising Effectiveness of Pre-Trained Vision Models for ControlPreprint
Parisi, S., Rajeswaran, A., Purushwalkam, S., & Gupta, A. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2203.03580 · Cited by 251 · cited 5×
“, 2022), DMControl from (Parisi et al., 2022), ImageNav from (Yadav et al.”
BackgroundMethodResultInfluentialSelf-citationLanguage-Driven Representation Learning for Robotics
Karamcheti, S., Nair, S., Chen, A. S., Kollar, T., Finn, C., Sadigh, D., & Liang, P. (2023) · Robotics: Science and Systems Conference · doi:10.48550/arxiv.2302.12766 · Cited by 240 · cited 2×
“[27] trains a ViT with a masked encoding objective on pairs of image frames and text.”
BackgroundLast Layer Re-Training is Sufficient for Robustness to Spurious CorrelationsPreprint
Kirichenko, P., Izmailov, P., & Wilson, A. G. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.02937 · Cited by 213
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects
Batra, D., Gokaslan, A., Kembhavi, A., Maksymets, O., Mottaghi, R., Savva, M., Toshev, A., & Wijmans, E. (2020) · arXiv · doi:10.48550/arxiv.2006.13171 · Cited by 177 · cited 2×
“We consider two tasks in Habitat: image-goal navigation ( ImageNav ) [44] and object-goal navigation ( ObjectNav ) [45].”
BackgroundMethodSelf-citationSimple but Effective: CLIP Embeddings for Embodied AI
Khandelwal, A., Weihs, L., Mottaghi, R., & Kembhavi, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01441 · Cited by 172 · cited 2×
“Inspired by the advancements in self-supervised learning, recent work has incorporated visual representation learning into the training pipelines for EAI agents (Parisi et al., 2022; Nair et al., 2022; Radosavovic et al., 2022; Ma et al., 2022; Khandelwal et al., 2022; Yadav et al., 2022b; 2023).”
BackgroundMethodRobot Learning in Homes: Improving Generalization and Reducing Dataset BiasPreprint
Gupta, A., Murali, A., Gandhi, D., & Pinto, L. (2018) · Neural Information Processing Systems · doi:10.48550/arxiv.1807.07049 · Cited by 170
Surgical Fine-Tuning Improves Adaptation to Distribution ShiftsPreprint
Lee, Y., Chen, A. S., Tajwar, F., Kumar, A., Yao, H., Liang, P., & Finn, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.11466 · Cited by 169
Baevski, A., Babu, A., Hsu, W.-N., & Auli, M. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2212.07525 · Cited by 166 · cited 7×
“(2020); Baevski et al. (2022b); Chen et al. (2020; 2021). These algorithms use contrastive Chen et al. (2020; 2021), distillation-based Caron et al. (2020); Baevski et al. (2022b), or reconstructive Bao et al. (2021); He et al. (2021) objectives for training. Recently, a flurry of works have proposed using the vision…”
BackgroundMethodInfluentialProcTHOR: Large-Scale Embodied AI Using Procedural Generation
Deitke, M., VanderBilt, E., Herrasti, A., Weihs, L., Salvador, J., Ehsani, K., Han, W., Kolve, E., Farhadi, A., Kembhavi, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.06994 · Cited by 155 · cited 2×
“Thesolutions,basedonautoregressivelanguagemodelinginGPT[47,48,4]and maskedautoencoding in BERT[14],areconceptuallysimple:theyremoveaportionofthedataandlearntopredicttheremovedcontent.”
BackgroundHabitat 2.0: Training home assistants to rearrange their habitat
Szot, A., Clegg, A., Undersander, E., Wijmans, E., Zhao, Y., Turner, J., Maestre, N., Mukadam, M., Chaplot, D., Maksymets, O., et al. (2021) · arXiv · doi:10.48550/arxiv.2106.14405 · Cited by 143 · cited 6×
“We curate C ORTEX B ENCH , a benchmark for evaluating PVRs, consisting of 17 tasks spanning low-level locomotion [10], table-top manipulation [11], dexterous manipulation [12], multi-finger coordinated manipulation [13], indoor visual navigation [14], and mobile manipulation [15].”
BackgroundMethodInfluentialSelf-citationSelf-supervised Pretraining of Visual Features in the Wild
Goyal, P., Caron, M., Lefaudeux, B., Xu, M., Wang, P., Pai, V., Singh, M., Liptchinsky, V., Misra, I., Joulin, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.01988 · Cited by 141 · cited 1×
“Several works have showed that scaling model and dataset size improves performance on vision tasks like image classification (Zhai et al., 2022; Tian et al., 2021; Goyal et al., 2021).”
BackgroundLife Ascending: The Ten Great Inventions of Evolution
Lane, N. (2009) · Choice Reviews Online · doi:10.5860/choice.47-4400 · Cited by 130
Finetune like you pretrain: Improved finetuning of zero-shot vision models
Goyal, S., Kumar, A., Garg, S., Kolter, J. Z., & Raghunathan, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01853 · Cited by 121
Offline Visual Representation Learning for Embodied NavigationPreprint
Yadav, K., Ramrakhya, R., Majumdar, A., Berges, V.-P., Kuhar, S., Batra, D., Baevski, A., & Maksymets, O. (2022) · arXiv.org · doi:10.48550/arxiv.2204.13226 · Cited by 118
Self-citationHabitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI
Ramakrishnan, S. K., Gokaslan, A., Wijmans, E., Maksymets, O., Clegg, A., Turner, J., Undersander, E., Galuba, W., Westbury, A., Chang, A. X., et al. (2021) · arXiv · doi:10.48550/arxiv.2109.08238 · Cited by 106 · cited 1×
“The dataset is comprised of 80 training, 20 validation, and 20 testing scenes and utilizes the Habitat simulator [15, 65] and HM3D [68] environments.”
MethodSelf-citationOVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavPreprint
Yadav, K., Majumdar, A., Ramrakhya, R., Yokoyama, N., Baevski, A., Kira, Z., Maksymets, O., & Batra, D. (2023) · arXiv.org · doi:10.48550/arxiv.2303.07798 · Cited by 103 · cited 1×
“, 2022), to selectively updating some or all weights of the backbone (Kumar et al., 2022; Hansen et al., 2022a; Yadav et al., 2023).”
BackgroundSelf-citationThe Surprising Effectiveness of Representation Learning for Visual Imitation
Pari, J., Shafiullah, N. M., Arunachalam, S. P., & Pinto, L. (2021) · arXiv · doi:10.48550/arxiv.2112.01511 · Cited by 94 · cited 1×
“By comparison, [41] find simple”
Habitat-Matterport 3D Semantics Dataset
Yadav, K., Ramrakhya, R., Ramakrishnan, S. K., Gervet, T., Turner, J., Gokaslan, A., Maestre, N., Chang, A. X., Batra, D., Savva, M., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00477 · Cited by 92 · cited 3×
“We present an evaluation of object navigation ( ObjectNav ) using the HM3D-S EM dataset [61].”
MethodSelf-citationMulti-skill Mobile Manipulation for Object RearrangementPreprint
Gu, J., Chaplot, D. S., Su, H., & Malik, J. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2209.02778 · Cited by 91 · cited 1×
“, 2021) includes a set of mobile manipulation tasks in which an agent controls a Fetch robot with a 7-DoF arm, mobile base (Gu et al., 2022), and suction gripper to rearrange objects in apartment scenes.”
BackgroundSelf-citationHabitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scale
Ramrakhya, R., Undersander, E., Batra, D., & Das, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00511 · Cited by 90
Self-citationOn Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline
Hansen, N., Yuan, Z.-C., Ze, Y., Mu, T.-Z., Rajeswaran, A., Su, H., Xu, H., & Wang, X.-L. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2212.05749 · Cited by 86
Self-citationTriFinger: An Open-Source Robot for Learning DexterityPreprint
Wüthrich, M., Widmaier, F., Grimminger, F., Akpo, J., Joshi, S., Agrawal, V., Hammoud, B., Khadiv, M., Bogdanovic, M., Berenz, V., et al. (2020) · Conference on Robot Learning · doi:10.48550/arxiv.2008.03596 · Cited by 86
MoDem: Accelerating Visual Model-Based Reinforcement Learning with DemonstrationsPreprint
Hansen, N., Lin, Y.-X., Su, H., Wang, X.-L., Kumar, V., & Rajeswaran, A. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2212.05698 · Cited by 79
Self-citationRoboHive: A Unified Framework for Robot LearningPreprint
Kumar, V., Shah, R. M., Zhou, G.-Y., Moens, V., Caggiano, V., Vakil, J., Gupta, A., & Rajeswaran, A. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2310.06828 · Cited by 73 · cited 1×
“We use Robohive [73] as the middleware to interface with all sensors and robots.”
MethodSelf-citationPIRLNav: Pretraining with Imitation and RL Finetuning for OBJECTNAV
Ramrakhya, R., Batra, D., Wijmans, E., & Das, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01716 · Cited by 71
Self-citationDivide and Contrast: Self-supervised Learning from Uncurated Data
Tian, Y., Hénaff, O. J., & van den Oord, A. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00991 · Cited by 69 · cited 1×
“Several works have showed that scaling model and dataset size improves performance on vision tasks like image classification (Zhai et al., 2022; Tian et al., 2021; Goyal et al., 2021).”
BackgroundRetrospectives on the Embodied AI WorkshopPreprint
Deitke, M., Batra, D., Bisk, Y., Campari, T., Chang, A. X., Chaplot, D. S., Chen, C., Pérez-D’Arpino, C., Ehsani, K., Farhadi, A., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.06849 · Cited by 62
Self-citationMemory-Augmented Reinforcement Learning for Image-Goal Navigation
Mezghan, L., Sukhbaatar, S., Lavril, T., Maksymets, O., Batra, D., Bojanowski, P., & Alahari, K. (2022) · 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981090 · Cited by 61 · cited 1×
“Our study conducts ImageNav experiments using the standard dataset presented in [64].”
MethodSelf-citationZero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation
Al-Halah, Z., Ramakrishnan, S. K., & Grauman, K. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01652 · Cited by 58 · cited 1×
“We train agents with the reward functions presented in [67] utilizing the following settings: success weighting c s = 5 .”
MethodInstruction-Following Agents with Multimodal TransformerPreprint
Líu, H., Lee, L., Lee, K., & Abbeel, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.13431 · Cited by 42 · cited 2×
“In [31], a multimodal transformer is pretrained on web-scale image-and-text data, and then used with a transformer-based policy for table-top manipulation tasks.”
MethodSelf-citationOpen-World Object Manipulation using Pre-trained Vision-Language Models
Stone, A., Xiao, T., Lu, Y., Gopalakrishnan, K., Lee, K.-H., Vuong, Q., Wohlhart, P., Kirmani, S., Zitkovich, B., Xia, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.00905 · Cited by 33 · cited 2×
“Suchmethodsincludecontrastivelearningwithdataaugmentation[25,26,27,28],dynamicsprediction[29,30],bi-simulation[31],temporalorgoaldistance[32,33],ordomainspecificinformation[34]. from 7 existing EAI…”
Pave the Way to Grasp Anything: Transferring Foundation Models for Universal Pick-Place Robots
Yang, J.-G., Tan, W., Jin, C.-H., Liu, B., Fu, J., Song, R.-H., & Wang, L.-M. (2023) · arXiv.org · doi:10.48550/arxiv.2306.05716 · Cited by 30 · cited 2×
“Suchmethodsincludecontrastivelearningwithdataaugmentation[25,26,27,28],dynamicsprediction[29,30],bi-simulation[31],temporalorgoaldistance[32,33],ordomainspecificinformation[34]. from 7 existing EAI…”
VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied RearrangementPreprint
Wijmans, E., Essa, I., & Batra, D. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2210.05064 · Cited by 27
Self-citationSchreiber, M. (1990) · doi:10.1097/00004424-199010000-00017 · Cited by 20
CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learning
Mandi, Z., Bharadhwaj, H., Moens, V., Song, S., Rajeswaran, A., & Kumar, V. (2022) · arXiv · doi:10.48550/arxiv.2212.05711 · Cited by 19 · cited 1×
“• The close-drawer and toaster plunge tasks are similar to the Cacti setup [72].”
MethodSelf-citationMasked Contrastive Representation LearningPreprint
Yao, Y., Desai, N., & Palaniswami, M. S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.06012 · Cited by 10 · cited 1×
“, 2021) with masked image modeling (He et al., 2021; Baevski et al., 2022a; Yao et al., 2022), which among other benefits reduces the computation time required for pre-training.”
Background
Cited by
Sorted by citations
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models
Zhao, Q., Lu, Y., Kim, M. J., Fu, Z., Zhang, Z., Wu, Y., Li, Z., Ma, Q., Han, S., Finn, C., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00166 · Cited by 231
“Some approaches incorporate VLMs [3, 44, 58] into end-to-end trainable policies by using them as pretrained backbone for better visuo-language representation [10, 26, 38, 43, 53].”
An Embodied Generalist Agent in 3D WorldPreprint
Huang, J., Yong, S.-L., Ma, X., Linghu, X., Li, P.-H., Wang, Y., Li, Q., Zhu, S.-C., Jia, B., & Huang, S.-Y. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2311.12871 · Cited by 475
HumanPlus: Humanoid Shadowing and Imitation from HumansPreprint
Fu, Z.-P., Zhao, Q., Wu, Q., Wetzstein, G., & Finn, C. (2024) · Conference on Robot Learning · doi:10.48550/arxiv.2406.10454 · Cited by 433
“Human data has been used extensively for robot learning, including for pre-training visual or intermediate representations or tasks [8, 28, 50, 53, 66, 79, 86, 99] leveraging Internet-scale data [18, 26, 27], and for imitation learning on in-domain human data [3, 12, 43, 47”
MethodA Survey on Vision--Language--Action Models for Embodied AI
Ma, Y., Song, Z., Zhuang, Y., Hao, J., & King, I. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3650584 · Cited by 403
“Pixel-level information contains rich details—including segmentation masks, object positions, and depth estimation—that are generally more useful for robot manipulation tasks requiring high precision, as demonstrated by VC-1’s comparison [32].”
Self-citationVideo Prediction Policy: A Generalist Robot Policy with Predictive Visual RepresentationsPreprint
Hu, Y., Guo, Y., Wang, P., Chen, X., Wang, Y., Zhang, J., Sreenath, K., Lu, C., & Chen, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.14803 · Cited by 257
“No-tably, methods like R3M (Nair et al., 2022), vip (Ma et al., 2022), VC-1 (Majumdar et al., 2023), and Voltron (Karam-cheti et al., 2023) have specifically focused on embodied tasks by innovating pre-training approaches on human manipulation video datasets (Goyal et al., 2017; Grauman et al., 2022).”
Influential
Metadata
- Authors
- Majumdar, A., Yadav, K., Arnaud, S., Ma, Y. J., Chen, C., Silwal, S., Jain, A., Berges, V.-P., Abbeel, P., Malik, J., Batra, D., Lin, Y., Maksymets, O., Rajeswaran, A., & Meier, F.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 655-677
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2303.18240
- OpenAlex
- W7133188016
- Cited by
- 49
- References
- 83
- Citations by year
- 2026: 12 · 2025: 14 · 2024: 21 · 2023: 2
- FWCI
- 25.21
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Keywords
- embodied AI, visual cortex, artificial intelligence, visual perception, neuromorphic computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
