Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?

“We choose the number of epochs per run such that the number of model updates remain constant across all runs and match the number of model updates taken by MAE on the ImageNet dataset.”

“VC-1 is superior to the second-best model ( Ego4D+MN ViT-L, Table 4 row 9), which has an average rank of 3.1.”

“3 However, curiously, the kinds of PVRs that are locally-dominant in CORTEXBENCH differ significantly in the size and type of pre-training datasets: CLIP (Radford et al., 2021) was pretrained on 400M image-text pairs from the web; MVP (Radosavovic et al.”

“These algorithms use contrastive [21, 22], distillation-based [19, 20], or reconstructive [18, 23] objectives for training.”

“Since VC-1 was trained with MAE (He et al., 2021), it captures features that are generally useful for reconstructing images.”

“In this work, we ask the same question that Fukushima [1, 2] asked nearly 50 years ago – how do we design an artificial visual cortex , the module in a computational system that (together with a policy) enables an agent to convert camera input into actions?”

“End-to-end (E2E) fine-tuning with a task-specific loss function can in-principle capture both of the aforementioned benefits of adaptation, and is widely used in computer vision literature (He et al., 2020; Caron et al., 2021; He et al., 2021; Baevski et al., 2022b).”

“We use a broad definition of adaptation [55], which, in the context of large pre-trained foundation models, can take several forms from simple prompting [56], to selectively updating some or all weights of the backbone [5, 35, 57].”

“Generic robot visual pretraining also improves down-stream manipulation [22, 23, 24, 25].”

“Early works mainly used egocentric corpora [47, 48] to learn robot visual representations [17, 49, 50].”

“The field has progressed from traditional discrete mapping and planning (Chaplot et al., 2020; Campos et al., 2021; Labb´e & Michaud, 2018) toward approaches that integrate more expressive visual representations (Khandelwal et al., 2022; Majumdar et al., 2023).”

“Beyond static images, researchers leverage large-scale video datasets [54], [55] and egocentric collections [56], [57] to train robust representations for robotic perception, exemplified by VIP [58] and VC-1 [59].”

“Inspired by the notion of selective imitation in psychology studies Brody & Stoneman (1981), SILO Lee et al. (2020) proposes a reinforcement learning framework to learn which timesteps of a demonstration should be imitated.”

“One line of research [30, 23, 28, 33] directly adopts 2D vision paradigms such as contrastive learning and masked image modeling.”

“Integrating vision foundation models as perceptual modules within robotic learning pipelines has yielded dramatic performance gains—not only in basic manipulation tasks but also in long-horizon planning and generalization [Majumdar et al., 2023].”

“Recent work integrates multiple modalities, combining vision with language [22, 36, 42, 50, 64], proprioception with vision [47, 62, 66], even richer [6, 53, 69].”

“For example, “ arti fi cial visual cortices ” mimic some aspects of biological visual processing where the learned representations can later be used to train robotic controllers (Nair et al, 2022; Majumdar et al, 2023).”

“…representation from collected data by applying different self-supervised methodologies, which can also be fine-tuned during policy learning (Goel et al., 2018; Anand et al., 2019; Kulkarni et al., 2019; Schwarzer et al., 2021; Xiao et al., 2022; Montalvo et al., 2023; Majumdar et al., 2023).”

“Many works [2, 6, 13] follow this framework.”

“[31], LVD : LVD-142M [27], K : Kinetics [20], E4D : Ego4D [12], E4D+MNI [23].”

“Leveraging large-scale human video datasets has been explored in robot learning to pretrain policies [77, 78, 79, 80, 8, 81, 82, 83, 84, 85] and augment RL [86, 87, 88].”

“…are: (1) CLIP (ViT-B/32 model) (Radford et al. (2021)), a model trained using contrastive learning on image-text pairs collected from the internet and (2) VC1 (ViT-B/32 model) (Majumdar et al. (2023)), a transformer-based encoder model pre-trained on data encompassing control and robotics tasks.”

“Robot foundation models [1, 2, 3, 4, 5, 6, 7, 8] leverage large-scale datasets spanning diverse objects, scenes, and embodiments to produce generalizable, cross-platform robot policies.”

“We only update the parameters of the transformer and projection layers while freezing the visual encoder since prior work shows this is an effective strategy for embodied AI (Khandelwal et al., 2022; Majumdar et al., 2023).”

“There are a plethora of techniques for pretraining visual representations using unsupervised learning objectives [38, 34, 30, 32, 41, 49, 47, 13].”

“Their components dated from 2019: encoder-only language modeling saw little follow-up after RoBERTa [16], and no pretrained visual representation of the time worked across embodied tasks [17].”

“VC-1 [Majumdar et al., 2023], R3M [Nair et al., 2022], V-JEPA 2 [Assran et al., 2025], VideoMAE [Tong et al., 2022], and DINOv2 [Oquab et al., 2024] are compared by downstream control success, which mixes perception, physical inference, and policy optimisation into one number.”

“The VC-1 study [103] systematically evaluates this SSL on CortexBench, a suite of 17 tasks covering locomotion, navigation, dexterous manipulation, and mobile manipulation.”

“Moving from pixels to latent codes reduces the burden of reconstruction, but compactness alone does not guarantee semantic abstraction or alignment with action-relevant dynamics (Majumdar et al. 2023).”

“Robotic manipulation has shifted from state-based reinforcement learning [19, 20], which often assumes compact low-dimensional states, to policies learned from high-dimensional visual observations [1, 4, 21, 22].”

“Vision provides dense spatial observations and can leverage powerful pretrained representations for robotic control (Radosavovic et al., 2023; Majumdar et al., 2023).”

“…studies masked visual pretraining for robot manipulation, while VC-1 evaluates a range of pretrained visual representations across robotic tasks and argues for large-scale visual pretraining as a reusable perceptual substrate for embodied control [Radosavovic et al., 2023, Majumdar et al., 2023].”

“Additionally, following the evaluation protocol from VC-1 [35] and PCM [36], we report two aggregate metrics: Mean S.R. and Mean Rank.”

“While robotics-specific self-supervised encoders exist [53, 54, 55, 56], their ability to quickly adapt to embodiments unseen during pretraining remains underexplored.”

“…setup allows us to test the practical effectiveness of object-centric representations under real-world embodiment constraints and visual noise. b) Pre-trained visual models: Based on recent state-of-the-art evaluations [2], [10], we selected 7 pre-trained visual models that we depict in Table I.”

“…degrade when the test scene differs from training, and common remedies such as domain randomization [30], broader demonstration coverage [9], and visual pretraining [31, 32] narrow but rarely close this gap, since their effectiveness is bounded by what the observation encoder can represent [33].”

“In the second stage, we freeze the control policy and train only a lightweight visual bridge that maps real observations to the policy’s expected input space [10], [11], [12].”

“With the maturation of the pre-trained visual representation learning paradigm, many studies [4], [17], [18], [19] focus on enhancing visual representations in robotics using pre-training models.”

“These approaches include pre-training latent visual representations [14, 38, 44, 57], inferring action trajectories or plans [3, 31, 46, 70, 78], learn-2 ing value or reward functions [9, 37], reconstructing human hand or hand-object interaction [40, 50, 53, 59, 63], parsing interaction goals and…”

“The models included in our study are ViT [41] (pre-trained with ImageNet-21k dataset), CLIP [32] (pre-trained with image-text pairs collected from the internet), VC-1 [34] (pre-trained with the Embodied AI dataset known as CORTEXBENCH), and ResNet [42] (pre-trained with ImageNet-1k dataset).”

“Visual Cortex Visual cortex (VC) [8] is a ViT based PVR model trained using Masked Auto-Encoding (MAE) on egocentric data, with the aim of being used for embodied intelligence.”

“As for self-supervised learning, a framework with far richer feedback, its recent reformulation has enabled unprecedented performance on multi-task problems with foundation models in language [39], [110], [111], [112], vision [36], [113], and robotics [114], [115], to name a few.”

“…and vision foundation models to enhance robotic capabilities, utilizing them as pretrained visual representations that accelerate robotic policy learning [29, 32, 30, 19, 31], for object localization in robotics tasks [9, 45], and for high-level planning and reasoning [1, 17, 42, 16, 43, 18, 6].”

“Following the approach from [22], we collect 64 steps of experience and subsequently perform 2 PPO epochs with a mini-batch size of 2 .”

“Early work learns visual representations from human videos for downstream robot control (Nair et al., 2022; Majumdar et al., 2023), while recent approaches incorporate human videos into robot foundation model training to learn transferable visual and behavioral knowledge (Yang et al., 2025; Luo et…”

“The R3M encoder used in our main experiments performs best, while VC-1 [67] is competitive and the general vision-language SigLIP2 [68] is weaker.”

“We consider six pretraining emphases: Visual self-supervision: VC-1 (Majumdar et al., 2023) and VideoMAE V2 (Wang et al., 2023) learn visual structure through masked image or video reconstruction.”

“Other work probes what visual representations encode about manipulation and physics (Majumdar et al., 2023; Burns et al., 2023; Yeom et al., 2026; Garrido et al., 2025; Joseph et al., 2026).”

“This experience has been used to learn reusable visual representations and latent plans for downstream robot control [8, 9, 10, 32, 33], while a complementary line transfers human motor behavior through pose reconstruction, action retargeting, portable motion capture, or human–robot co-training…”

“Whether egocentric human video transfers to downstream embodied and video-language tasks is a question prior work has already answered at scale [11, 15, 16, 22]; what remains unsolved is how to capture such video, with calibrated geometry and time-aligned inertial data, at a cost that permits…”

“Visual representation learning has provided strong spatial and semantic features for robot learning through language supervision, masked reconstruction, segmentation, and robotics-oriented pre-training on human or embodied video [5, 6, 7, 8, 9, 10, 11, 12].”

“We compare CAIP against a representative set of vision encoders spanning self-supervised, language-supervised, video, and robotics-pretrained representations: R3M [15], MVP [16], DINOv2 [4], SigLIP [2], SigLIP 2 [17], VideoMAE [21], VC-1 [22], and the native Qwen3.”

“We include three representative single-task settings: • VC-1 [14]: 14 tasks from four simulators — Adroit (AD) [10], Meta-World (MW) [28], DMControl (DMC) [22], and TriFinger (TF) [25].”

“Su-pervised methods [25, 28, 39, 48] typically rely on large-scale annotated 2D image or video datasets [6, 17, 18, 27].”

“VC-1 (Majumdar et al., 2023) is trained on egocentric and third-person videos.”

“Task-driven suites such as VC-1 [31] and SPA [19] utilize the same encoders in navigation and object-search policies, measuring downstream success and thereby tying representation quality to action.”

“As Embodied Artificial Intelligence (EAI) [32, 43] becomes more integrated into daily life, the study of egocentric video understanding [40, 49] has grown increasingly prominent.”

“A powerful vision encoder can also be pretrained via masked prediction without annotations or labels [14], [15], [16], and the encoder can be used to solve various vision tasks [17], [18].”

“Enabling machines to perceive and interact in real-world scenarios through vision and language has been a relentless pursuit in embodied artificial intelligence [4, 17, 26].”

“These models introduce video representation learning [33], multi-agent dynamics, and forward/inverse prediction, yet all remain black-box latent estimators without explicit guarantees of physical consistency.”

“Among these models, VC-1 is an MAE [34] trained on combinations of egocentric and third-person videos.”

“To demonstrate the effectiveness of H-InDex, we evaluate diverse recent strong visual representations for motor control, including (i) VC-1 [17] 1 , which trains masked auto-encoders over 5.6M images with over 10,000 GPU-hours and we use the ViT-B [4] model (86M parameters); (ii) MVP [23,31], which…”

“Additionally, vision foundation models have been used to extract semantic and structural representations from static images [31, 32, 33], as demonstrated in MT-R3M [34], which combines R3M embeddings with a Transformer-based policy head.”

“Human demonstrations have been widely used to scale robot learning, with early works leveraging human videos primarily for representation learning or intent inference [17, 36, 15, 14, 45].”

“These objectives provide strong semantic priors, but can also encourage invariance to small object offsets, local boundaries, and pre-contact alignments that precision manipulation must react to [6, 7].”

“There are also other people studying the benefit of pretrained visual models for visuomotor tasks (Shah & Kumar, 2021; Majumdar et al., 2023; Hansen et al., 2023b; Parisi et al., 2022) while others try to train large policy networks directly with transformers (Vaswani et al., 2017) and huge…”

“To leverage vision foundation models (VFMs), one line of work directly adopts off-the-shelf foundation encoders such as DINOv2 [39] or CLIP [42] as frozen perception backbones, or propose robot-centric pretraining [35, 37, 43] and distillation methods [50] from large-scale data [11, 16].”

“Earlier work learned reusable visual representations from diverse human video [3], [4], [13].”

“These applications motivate evaluating what information is accessible in large-scale pretrained visual representations and how adaptation affects their usefulness [14].”

“Vision-language navigation (VLN) is a fundamental task in embodied AI, requiring an embodied agent to follow language instructions through 3D environments (Anderson et al., 2018b; Krantz et al., 2020; Majumdar et al., 2023).”

“Many robotics studies pretrain visual representations to improve task performance [32, 34, 40].”

“Which pretrained representation to hand a policy is a long-standing question in robot learning [37, 41, 47].”

“Several works build upon RGB vision foundation models [11]–[13], keeping encoders frozen or lightly fine-tuned for point-goal navigation [1], [2], image-goal navigation [9], [14], and visual affordance prediction [15]–[17].”

“Comparative studies find no single dominant representation [48], show that general vision datasets can match egocentric video for visuomotor pretraining [19], and note that domain-specific pretraining does not guarantee robustness under visual shifts [11].”

“…specifically for manipulation from human video, masked image modeling, or sensorimotor prediction (Nair et al., 2023; Radosavovic et al., 2023; Majumdar et al., 2023), and finds that the benefit of pretrained features varies substantially across tasks and evaluation settings (Burns et al.,…”

“In embodied learning, these two standard encoders represent different pretraining paths (Majumdar et al., 2023; Jing et al., 2023).”

“Pretrained representations [10], [11] and domain randomization [12] support visual transfer, yet history still mixes task evidence with appearance and camera motion.”

“R3M [25] and VC-1 [26] learn visual representations without action supervision, transferring knowledge through the perceptual encoder.”

“Frozen SigLIP 2, VC-1, R3M, and DINOv3 features exhibit the same type of separation (Tschannen et al., 2025; Majumdar et al., 2023; Nair et al., 2023; Siméoni et al., 2025); the separation is therefore not specific to VLA architectures.”

“We evaluate frozen visual representations for embodied control on CortexBench [59], using two dexterous-manipulation tasks from Adroit [60] ( pen , relocate ) and five tabletop tasks from Meta-World [61] ( assembly , bin-picking , button-press-topdown , drawer-open , and hammer ).”

“Prior works [29][30][31][32] train a visual encoder using both human and robot video data to avoid learning the policy network from scratch.”

“Baselines include pretrained 2D features [10], [11], action diffusion [15], VLAs [38], [39], and adapted 3D policies [1]– [3].”

“General-purpose visual encoders, including R3M [23], VIP [24], and VC-1 [25], learn reusable representations from large-scale datasets but are not designed specifically for the fine-grained spatial reasoning required in submillimeter assembly.”

“Modern visual backbones provide strong semantic and spatial features (Oquab et al., 2024; Zhai et al., 2023), yet their pretraining objectives are not designed to preserve control-critical variables such as object pose, gripper–object distance, contact phase, and occlusion (Majumdar et al., 2023).”

“With the growth of robot datasets[5], [6], such pretrained models have been further adapted using large-scale robot data, enabling their semantic priors to transfer to embodied control[17], [18].”

“In practice, though, the choice of which layer to start from is either an expensive sweep kept only for its best result, or a fixed truncation depth used by convention, especially in robotics (Nair et al., 2022; Majumdar et al., 2023).”

“Early approaches focused on learning visual representations through self-supervised learning [17], [16], [15], [23], [24].”