Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?

“We choose the number of epochs per run such that the number of model updates remain constant across all runs and match the number of model updates taken by MAE on the ImageNet dataset.”

“VC-1 is superior to the second-best model ( Ego4D+MN ViT-L, Table 4 row 9), which has an average rank of 3.1.”

“3 However, curiously, the kinds of PVRs that are locally-dominant in CORTEXBENCH differ significantly in the size and type of pre-training datasets: CLIP (Radford et al., 2021) was pretrained on 400M image-text pairs from the web; MVP (Radosavovic et al.”

“These algorithms use contrastive [21, 22], distillation-based [19, 20], or reconstructive [18, 23] objectives for training.”

“Since VC-1 was trained with MAE (He et al., 2021), it captures features that are generally useful for reconstructing images.”

“In this work, we ask the same question that Fukushima [1, 2] asked nearly 50 years ago – how do we design an artificial visual cortex , the module in a computational system that (together with a policy) enables an agent to convert camera input into actions?”

“End-to-end (E2E) fine-tuning with a task-specific loss function can in-principle capture both of the aforementioned benefits of adaptation, and is widely used in computer vision literature (He et al., 2020; Caron et al., 2021; He et al., 2021; Baevski et al., 2022b).”

“We use a broad definition of adaptation [55], which, in the context of large pre-trained foundation models, can take several forms from simple prompting [56], to selectively updating some or all weights of the backbone [5, 35, 57].”

“…visual features learned from large-scale human video can improve downstream manipulation performance [6], while masked visual pretraining and subsequent empirical studies further confirmed that stronger pretrained visual encoders can improve transfer and robustness in embodied settings [7], [8].”

“View-Invariant Representations from RGB. Pre-trained vision encoders have become standard backbones for imitation learning policies [16, 17, 18], inheriting the broad capabilities of general-purpose vision foundation models [19, 20, 1].”

“Frozen pretrained vision models can be competitive representations for control [16], and R3M [17] and VC-1 [18] study this pattern in robot learning.”

“Building on this foundation, large-scale visual foundation models for embodied agents [9] were introduced to improve generalization across tasks and platforms.”

“Recent constraint-grounded methods [63, 64] and representation backbones [65, 66, 67, 68, 69] similarly focus on better perception or abstract constraint satisfaction rather than explicit equations of motion.”

“…image segmentation bridges the gap between natural language understanding and fine-grained vision perception, which has wide applications in the fields of image-text retrieval [1], [2], [3], human-computer interaction [4], autonomous driving [5], [6], [7], embodied intelligence [8], [9] and so on.”

“Related lines include visual foundation models for embodied intelligence such as VC-1 [20], multi-agent coordination via retrieved skills [21], and autonomous-driving VLAs such as EMMA [22] and Alpamayo-R1 [23].”

“Concurrently, approaches such as MVP [12] and VC-1 [32] adopted Masked Autoencoders (MAE) [33] to reconstruct masked image patches, fostering strong local structural awareness and pixel-level understanding.”

“R3M [18] and VLP [16] proposed methods to transfer representations pretrained on large-scale vision-language datasets to downstream robotic tasks.”

“Several works have explored objectives like contrastive learning [41] and masked autoen-coding [36] to learn general-purpose visual encoders from common egocentric video datasets such as Ego4D [15] and Epic Kitchens [12, 13].”

“Benefiting from the increasing accuracy of visual encoders that help perceive target poses and environmental states [2-4] and the increasingly powerful information processing and logical reasoning capabilities of LLMs [1][5], VLAs can handle multi-process tasks in complex environments.”

“ORCA, on the other hand, simultaneously learns task prompts and visual prompts from the current visual state to derive task-specific conditions during the downstream training, without the need for additional optimization as of in TADP [32].”

“Experimental re-sults show that our method outperforms existing 2D and 3D approaches (Majumdar et al. 2023; Nair et al. 2022; He et al. 2016; Banino, Balaguer, and Blundell 2021; Qi et al. 2017; Ze et al. 2024) on benchmarks such as ManiSkill and RL-Bench (Gu et al. 2023; James et al. 2020).”

“For CLIP, e t is the output for the [CLS] token, following [8].”

“…between robot and human states (Luo et al., 2024), but the final goal should remain to exploit existing semantic understanding in pre-trained vision-language and other foundation models (Stone et al., 2023; Yuan et al., 2024; Zitkovich et al., 2023; Wulfmeier et al., 2023; Majumdar et al., 2023).”

“Similarly, Meta AI R3M [18] and VC-1 [19] used pre-trained visual representations to train a model and achieved average success rates of about 60% and 70% respectively as reported.”

“Facing data scale challenges, the field of robotic manipulation has begun to explore self-supervised pre-training as a means of reducing reliance on large labeled datasets [24], [25], [26], [27], [28], [29].”

“Furthermore, VC-1 [29] leverages MAE and has built a substantial dataset tailored for multiple embodied intelligence tasks.”

“Various works for quickly adapting a model to different environment features have been presented; e.g., for task difficulty levels [4, 15, 18, 30, 35], unseen tasks [9, 10, 20, 22, 40], and embodied properties [25, 27].”

“We plan to study large-scale pretraining for more complex tasks in the future.”

“This is in line with recent findings on the role of datasets in training powerful visual backbones [Majumdar et al., 2023].”

“The motivation is that although general vision models are making tremendous progress [24], robotic tasks are still very complex, and fine-tuning on a targeted dataset typically improves the performance of vision models [21] for robotics manipulation.”

“Many methods (Hafner et al., 2018; Majumdar et al., 2023; Lee et al., 2019a; Srinivas et al., 2020; Schwarzer et al., 2020) have shown the benefits of auto-encoders (Finn et al., 2015; Tschan-nen et al., 2018) to visual RL.”

“The robot planned a global path using A ∗ and followed this path using DWA [30] as the local planner.”