Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?

“We choose the number of epochs per run such that the number of model updates remain constant across all runs and match the number of model updates taken by MAE on the ImageNet dataset.”

“VC-1 is superior to the second-best model ( Ego4D+MN ViT-L, Table 4 row 9), which has an average rank of 3.1.”

“3 However, curiously, the kinds of PVRs that are locally-dominant in CORTEXBENCH differ significantly in the size and type of pre-training datasets: CLIP (Radford et al., 2021) was pretrained on 400M image-text pairs from the web; MVP (Radosavovic et al.”

“These algorithms use contrastive [21, 22], distillation-based [19, 20], or reconstructive [18, 23] objectives for training.”

“Since VC-1 was trained with MAE (He et al., 2021), it captures features that are generally useful for reconstructing images.”

“In this work, we ask the same question that Fukushima [1, 2] asked nearly 50 years ago – how do we design an artificial visual cortex , the module in a computational system that (together with a policy) enables an agent to convert camera input into actions?”

“End-to-end (E2E) fine-tuning with a task-specific loss function can in-principle capture both of the aforementioned benefits of adaptation, and is widely used in computer vision literature (He et al., 2020; Caron et al., 2021; He et al., 2021; Baevski et al., 2022b).”

“We use a broad definition of adaptation [55], which, in the context of large pre-trained foundation models, can take several forms from simple prompting [56], to selectively updating some or all weights of the backbone [5, 35, 57].”

“Most work are based on end-to-end training (Zhu et al., 2017; Mezghani et al., 2022; Majumdar et al., 2023), potentially supported through self-supervised losses (Majumdar et al., 2022b).”

“…physical intelligence (Hermann et al., 2023), where large-scale data collection is feasible but yields significantly smaller performance gains compared to language, as seen with existing Vision Transformers (Bear* et al., 2021; Nayebi et al., 2023; Bear et al., 2023; Majumdar et al., 2023).”

“Our work builds on top of the literature on pre-trained visual representations for control [6], [8], [9], [30]–[32].”

“Specifically, the robotics community has trended towards learning representations us-ing state-of-the-art self-supervised vision algorithms on diverse interaction datasets [8, 17, 39], and finetune the network on robotics tasks [9, 28–30, 33, 37, 42].”

“Consequently, various efforts have been made to pre-train on large-scale out-of-domain data for visuo-motor control [25, 26, 27, 8, 28, 29, 30, 31, 32].”

“We largely adhere to the original parameter settings from the CortexBench paper (Majumdar et al., 2023).”

“To scale this up, recent works like R3M [24], LIV [19], and VC-1 [20] train on diverse and large-scale human video data to learn useful visual task representations for robotic tasks.”

“We can also finetune the pretrained representation on the downstream task demo dataset [29] to make the retrieval more robust.”

“Conversely, approaches that encode detailed state information from raw images [5, 45] or dense features [26, 32, 34, 40, 49, 55] are often too high-dimensional and redundant to effectively represent motion via simple differences.”

“In the literature of robotics, numerous studies focused on developing generalized visual encoders for robotic agents across various domains [29, 30], exploiting pretrained visual encoders [31, 32], and establishing robust control policies with domain randomized techniques [33, 34].”

“However, pretrained visual representations such as VC-1 [56] have been used in this way, and therefore we use these methods as another point of comparison.”

“Notably, pre-trained visual representations [32, 46, 55, 56] have shown substantial promise in improving performance on the ImageNav task.”

“Implicit methods [31, 32, 35, 40] utilize large-scale contrastive learning to derive meaningful features from egocentric interaction data.”

“3) Baselines: To assess GP3’s contribution, we benchmark it against 6 representative methods, grouped in 3 categories: (1) 2D Robotic Representation Methods : 2D foundation model CLIP (ViT-B/16) [27] and the 2D pretrained policy backbones R3M [29] and VC-1 [28].”

“VC-1 [19] scales up the model size and dataset in MVP. Recently, learning visuomotor policies from point clouds has shown great promise [37, 44, 55, 57], but a universal pre-training paradigm for robotic point cloud data remains unexplored.”

“To manage this complex sparse-reward task, existing solutions [7] divide the problem into sub-tasks and design modular pipelines that use open-vocabulary object detectors such as Detic [28].”

“Large-scale robotic datasets [3, 69] have enabled the emergence of generalist, end-to-end robot foundation models [4, 2, 70, 71, 72, 73] which typically combine a vision-language model with a behavior cloning architecture [46, 47, 74, 11] to generate robot policies from a larger representation…”

“We hypothesize that VC-1 Large does not perform as well as VC-1 Base due to the limited amount of data our policies were trained with, and this would be consistent with the findings in[8], in which scaling data and model size does not always lead to better performance.”

“VC-1 [224] and R3M [112] show that large-scale egocentric pretraining can yield general visual backbones and support few-shot real-robot learning, while Voltron [225] and HRP [226] further incorporate language groundingandaffordancepriorstoimprovehigh-levelsemantics…”

“We evaluate Pixio on CortexBench [43], using four benchmarks: Adroit, DMC, MetaWorld, and Trifinger.”

“Instead of auxiliary representation learning, prior work has considered using pretrained visual representations for BC. Utilizing pretrained BC has been a reliable method to efficiently learn policies and improve generalization [42, 33, 37].”

“For all methods in Language Rearrangement, we use the pre-trained VisualCortex model (VC1, Majumdar et al., 2023) a ViT backbone designed for egocentric visual tasks.”

“Future works could leverage improvements in RL architectures and hyper-parameter tuning [5] and in the pre-training of visual backbones [70], [71], [72].”

“In fact, the original Octo paper has reported basic fine-tuning experiments, comparing its performance with [27] and a transformer-based model trained from scratch across four tasks.”

“Recently, it has been dominated by unsupervised and self-supervised methods [13, 25, 23, 12, 24, 40, 35, 9, 1, 28, 39, 26, 36].”

“Prior works [16][17][18][19] train a visual encoder using both human and robot video data to avoid learning the policy network from scratch.”

“Combined with traditional 3D visual grasping methods, the use of pre-trained models enhances the ability of visual–language-guided robotic grasping.”

“We use the front camera of 224 × 224 resolution without depth for the CortexBench and RLBench.”

“…work should build on our research to explore models that integrate compression-based reconstruction error with different forms of encoding sequential inputs, including using recent advances in neural network models for encoding of videos and events 64 , and text-based natural language input 65 .”

“Since then, several other works [17]–[20] have followed suit, demonstrating gains from various training objectives on robotics-relevant data [30], [31], including objectives specifically motivated by usage for control tasks [17], [18].”

“The paradigm shift driven by foundation models [8] is revolutionizing the communities who study sequential decision-making problems [80], with innovations focusing on control [2, 45, 38, 9], planning [76, 32, 33, 78, 17], pre-trained visual representation [57, 50, 67, 51], among others.”

“unification in unified models can come in the form of a single model for multiple tasks [50, 44, 3], capable of handling multiple modalities [2, 9, 5], or being learned from diverse domains [1, 51].”

“These systems exhibit human-like physical actions and continuously refine their strategies based on observation, action, and feedback, enabling them to handle complex, real-world tasks [64].”

“…employ foundation models from other domains as vision encoders (Parisi et al., 2022; Yuan et al., 2022), such as CLIP (Radford et al., 2021) and DINOv2 (Oquab et al., 2024), while others pretrain visual representations on egocentric video datasets (Nair et al., 2023; Majumdar et al., 2024).”

“Single-task policy — following [20], the policy π is implemented as an MLP predicting actions from the input which is a concatenation of the current frame, two frame differences and the proprioception input p t , ˆ a t = π r t, 1 − r t, 0 , r t, 2 − r t, 1 , r t, 2 , p t ; θ π , where [ ] is the…”

“Efforts such as R3M (Nair et al., 2022), VIP (Ma et al., 2023), MVP (Xiao et al., 2022), PIE-G (Yuan et al., 2022), and VC-1 (Majumdar et al., 2023) highlight this direction.”

“The sim2real gap can compromise strong performance achieved in simulation when agents are tested in the real-world [22, 29], as perception and control policies often do not generalize well to real robots due to in-accuracies in modelling, simplifications and biases.”

“Earlier paradigms leveraged such videos primarily for representation or visual reward learning [30, 31, 32, 33, 34, 35, 36, 37]; while these methods extract strong visual priors, they still rely heavily on downstream robot demonstrations to map perception to motor control.”

“The features learned by early RGB VFMs are often insufficient for robotics, which requires grounded, action-relevant features emphasizing geometry and physical affordances over general semantics [54].”

“E XT O BJ N AV : The Extended Object Navigation (E XT O BJ N AV ) task, first introduced in [11], builds on the O BJECT N AV task commonly used in embodied AI research [1, 20].”

“We freeze the pretrained models and use a 3-layer MLP as the policy network, training each task with 100 demonstrations following [34].”

“VC-1 [6] combines masked autoencoding with mixed-domain training (ImageNet and robot video), while Theia [8] distills multiple vision foundation models (e.g., CLIP, SAM, DINOv2) into a compact transformer backbone optimized for robotic policy learning.”

“VC-1 (VisualCortex-1) [35] aims to support a diverse range of sensorimotor skills and environments by training them on 5.6 million images of the Ego4D-MNI dataset consisting of the original Ego4D data [7] and additional data from manipulation tasks, navigation tasks and the ImageNet dataset [6].”

“Many methods (Hafner et al., 2018; Majumdar et al., 2023; Lee et al., 2019a; Srinivas et al., 2020; Schwarzer et al., 2020) have shown the benefits of auto-encoders (Finn et al., 2015; Tschan-nen et al., 2018) to visual RL.”

“With different policy heads such as ViLT (Liu et al., 2023; Kim et al., 2021), flow-matching and diffusion policy, we evaluate VER against pretrained vision encoders, including VC-1 (Majumdar et al., 2023), R3M (Nair et al., 2023), MVP (Xiao et al., 2022), RADIO (Ranzinger et al., 2024), VIP (Ma et al.)”

“More importantly, we also investigate methods beyond MAE [29], which has been the predominant choice in many existing works [19, 45, 55, 62].”

“Other works maximize the mutual information between the latents and control relevant variables such as the reward or actions [6, 14] or use masked auto-encoding [35, 42].”

“Finally, we show prediction in the reverse direction: recent video encoders (e.g. Video-MAE and VC-1 [1, 12]) more effectively predict fMRI brain activity from videos compared to image encoders.”

“Transfer is typically achieved by utilizing intermediate representations, either implicitly or explicitly: Using implicit visual representations [36], downstream policy learning [8, 9, 10, 11] has demonstrated visual generalization; however it has yet to show generalization to new actions.”

“On one hand, encoding knowledge in natural language as feature inputs for the control policy may hinder the control policy from fully recognizing the physical concepts underlying the knowledge [5, 28].”

“Specifically, agents capable of translating visual inputs into actions can be regarded as embodied systems [3], prompting NLP researchers to focus on constructing multi-modal LLM.”

“However, as concluded in [34], [35], it is evident that pre-training models exhibit variations in performance across diverse tasks.”

“Several works leverage egocentric human videos to learn visual or visuomotor representations [15, 16, 17, 18].”

“A separate line of work develops pre-training objectives specifically for robotic representations, ranging from single-modality self-supervised objectives [53, 36, 35, 47] to multimodal alignment with language, action, or robot proprioception [37, 34, 24, 52].”

“Several approaches leverage large-scale human video datasets to pretrain visual representations [68, 64, 63, 65], learn object and scene-level affordances [2, 11, 87], or predict robot actions [3, 5, 86].”

“To address this, we depart from works that naively extract the “CLS”-token from the high-dimensional embeddings, analogous to the Vision Transformer [21], [29], [49], [50] and, instead, propose an asymmetric latent bridging strategy between actor and critic to obtain a compact representation.”

“Considerable efforts have been made to develop visual encoders that embed RGB frames into compact latent vector spaces [26, 27, 28, 29, 30].”

“Before input into the sequence encoder, the patch tokens of each image are compressed into a one-dimensional vector with a 2D convolution layer followed by flattening, similar to [31].”

“We investigate two IDM implementations; one that follows prior work (Luo et al., 2025) that implements the IDM as a MLP network that takes in the outputs of a VC-1 (Majumdar et al., 2023) encoder model, finetuned on in-domain demonstrations.”

“To enable a comprehensive comparison, we benchmark against several SOTA VFMs with diverse design principles: DINOv2-Large [28] (vision-centric), SigLIP-Large [45] (language-contrastive), R3M-Resnet50 [26], VC-1 [25], and SPA [49] (embodied-specific).”

“While primarily focused in the manipulation setting, Chiang et al. (2024); Shah et al. (2023a,b,c) investigate foundation models for navigation, and the promise of cross-embodiment positive transfer inspires work which tackles both (Doshi et al., 2024; Majumdar et al., 2023; Yang et al., 2024a).”

“8 presents the inference time comparisons between our model and VC-1 [26], which is a ViT-based foundation model designed for behavior cloning of continuous control tasks.”

“We also evaluate state-of-the-art pre-trained representations from R3M [8] (Resnet-50), MVP [9] (ViT), VC-1 [10] (ViT), and Soup [12] (ViT), which all have been shown to improve IL and control performance.”

“The supported EAI tasks for this processor are selected from CortexBench [8].”

“This perspective is complementary to recent work on visual foundation models for embodied AI, which study “artificial visual cortex” representations reusable across tasks and embodiments [13].”

“The first type of work is rule-based planning methods using GS maps to exploit their geometric consistency and rendering quality [21], [22].”

“One particularly expensive and influential component of this process is the choice of visual representation, which modern generalist policies heavily rely on (Majumdar et al., 2023a).”

“IV-D, we chose to perform our main experiments with the MLP-BC policy using the Theia-Base [38] vision encoder.”

“Compared to producing pretrained visual representations from large-scale human videos [43], [44], [45], [46], extracting explicit affordances related to human hands enables more straightforward human-robot policy transfer.”

“…however, the adoption of pre-trained visual representations (PVRs) ( a.k.a. vision foundation models), i.e., computer vision models trained on large and diverse visual datasets, has emerged as a compelling alternative, moving away from the tabula-rasa approach [41], [25], [32], [44], [12], [21].”

“…0.415 0.061 0.262 0.178 0.113 0.362 ViP [28] self-supervised image non-robot 0.110 0.376 0.119 0.444 0.106 0.399 0.065 0.345 0.192 0.118 0.391 VC-1 [29] self-supervised image non-robot 0.101 0.372 0.104 0.385 0.080 0.326 0.0584 0.249 0.139 0.096 0.333 Octo [8] supervised image robot 0.147 0.415…”

“We evaluate several popular PVRs which are trained using different datasets, objectives, and methods: VC-1 [22], a Vision Transformer (ViT), was trained on a union of robotic and natural images using masked autoencoding (MAE).”

“Consequently, many studies have begun to explore the use of large-scale, readily available, out-of-domain human action video data to learn generalizable representations that can be transferred to robotic tasks [37, 27, 35, 30, 19, 26, 28, 42, 44, 23].”

“Several projects have trained vision encoders with internet datasets to accelerate robot learning [23]–[26].”

“This suggests that a well-pre-trained model can yield versatile representations applicable to a wide range of scenarios, not limited to specific adaptation algorithms (Parisi et al., 2022; Majumdar et al., 2023a).”

“Although some promising results have been obtained in recent works (e.g., R3M [2] and VC-1 [3]), they are still far from working the best in all general tasks and fine tuning based on Pre-trained Visual Representations (PVRs) is needed.”

“For instance, R3M [38] learns universal embodied representations from large-scale human video data using contrastive learning, VIP [39] produces dense reward functions for unseen robotic tasks, and MVP [40], VC-1 [41], and Voltron [42] explore MAE-style pretraining for robotic perception.”

“Previous approaches have attempted to bridge this gap through reinforcement learning with hand-crafted reward functions [1], [2], meta-learning for one-shot imitation [3], or domain alignment techniques between human and robot embodiments [4].”

“At the same time, cross-domain pretraining and self-improvement pipelines such as VC-1[50] and RoboCat [51] aim to generalize across tasks and embodiments via large-scale pretraining and autonomous data collection at the cost of considerable compute and data requirements depending upon robot morphology and sensor conf…”

“MAE [21] has been recognized for its strong localization capability, leading several studies [35, 10, 42] to adopt the recipe.”

“For example, through pre-training and time-contrastive learning over human ego-centric datasets [23,24,26], distilling knowledge from pre-trained computer vision (CV) models [31], and leveraging successful learning scheme of the CV field such as masked autoencoding [29].”

“First (when to adapt) , on-demand TTA triggers adaptation only when a substantial domain shift is detected and model performance drops.”

“3 we compare RANa ImageNav with state-of-the-art (SotA) models that typically finetune (with a navigation loss) visual encoders based on CLIP [47], ViT [49, 79, 80] and geometric FM [6] used for observation-goal comparison.”

“We adopt the training data and settings of [13] with five manipulation tasks, including Assembly , Button (button pressing), Hammer (hammering), Bin (picking-and-placing) and Drawer (drawer opening), for multi-task training (See examples in Fig.”

“Prior methods have explored initializing vision encoders from computer vision datasets [85, 58, 57, 18], or leveraging off-the-shelf task planners [38, 48, 73, 81].”