Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?

“We choose the number of epochs per run such that the number of model updates remain constant across all runs and match the number of model updates taken by MAE on the ImageNet dataset.”

“VC-1 is superior to the second-best model ( Ego4D+MN ViT-L, Table 4 row 9), which has an average rank of 3.1.”

“3 However, curiously, the kinds of PVRs that are locally-dominant in CORTEXBENCH differ significantly in the size and type of pre-training datasets: CLIP (Radford et al., 2021) was pretrained on 400M image-text pairs from the web; MVP (Radosavovic et al.”

“These algorithms use contrastive [21, 22], distillation-based [19, 20], or reconstructive [18, 23] objectives for training.”

“Since VC-1 was trained with MAE (He et al., 2021), it captures features that are generally useful for reconstructing images.”

“In this work, we ask the same question that Fukushima [1, 2] asked nearly 50 years ago – how do we design an artificial visual cortex , the module in a computational system that (together with a policy) enables an agent to convert camera input into actions?”

“End-to-end (E2E) fine-tuning with a task-specific loss function can in-principle capture both of the aforementioned benefits of adaptation, and is widely used in computer vision literature (He et al., 2020; Caron et al., 2021; He et al., 2021; Baevski et al., 2022b).”

“We use a broad definition of adaptation [55], which, in the context of large pre-trained foundation models, can take several forms from simple prompting [56], to selectively updating some or all weights of the backbone [5, 35, 57].”

“Some approaches incorporate VLMs [3, 44, 58] into end-to-end trainable policies by using them as pretrained backbone for better visuo-language representation [10, 26, 38, 43, 53].”

“Human data has been used extensively for robot learning, including for pre-training visual or intermediate representations or tasks [8, 28, 50, 53, 66, 79, 86, 99] leveraging Internet-scale data [18, 26, 27], and for imitation learning on in-domain human data [3, 12, 43, 47”

“Pixel-level information contains rich details—including segmentation masks, object positions, and depth estimation—that are generally more useful for robot manipulation tasks requiring high precision, as demonstrated by VC-1’s comparison [32].”

“No-tably, methods like R3M (Nair et al., 2022), vip (Ma et al., 2022), VC-1 (Majumdar et al., 2023), and Voltron (Karam-cheti et al., 2023) have specifically focused on embodied tasks by innovating pre-training approaches on human manipulation video datasets (Goyal et al., 2017; Grauman et al., 2022).”

“Our work is related to the literature on pre-training representations for control (Shah & Kumar, 2021; Cui et al., 2022; Parisi et al., 2022; Nair et al., 2022b; Xiao et al., 2022; Ma et al., 2022b; Fan et al., 2022; Majumdar et al., 2023).”

“Most of these works focus on single-task settings [35], [36], [39], [40], or simulated robot environments [37], [40].”

“Most works focus on pre-trained visual representations that can be used to initialize the encoder of the robot’s camera observations, either via supervised ImageNet classification [53], data augmentation [54, 55, 56, 57] or objectives that are tailored towards robotic control [58, 59, 60, 61, 62].”

“To enable generalization, many recent works propose using pre-trained visual representations trained primarily on non-robot datasets [31, 32], for learning manipulation policies [5, 6, 6–9 , 33– 36 ].”

“Previous works have made significant progress in pre-training only the vision part of the policy on human videos [48, 53, 34, 61] and pre-training the full policy [6, 14, 55] with a unified model and dataset format (e.g. using languages [5]).”

“A large body of work [10, 23, 24, 25] has studied leveraging cross-embodiment data to learn robotic policies.”

“One common way of using data beyond robot interactions for efficient learning is to pre-train the visual representations which serve as backbones for the policy models [62, 63, 64, 65, 66] with passive human videos [23, 67] and image data [68].”

“A key area is learning universal vision representations or world-models, typically trained on large, diverse offline datasets using either masking strategies [22, 17, 12, 67, 50, 35] or contrastive objectives [14, 41, 52, 68, 40, 25, 2, 34, 46].”

“Motivated by egocentric vision, recent studies [60, 61, 62, 63, 64, 65, 66] pretrain representations using human videos, showing positive transfer.”

“Many works employ vision models, pre-trained on large-scale image datasets, to generate visual representations for visuomotor control tasks [21], [31]–[34].”

“We compare the effects of using our own visual encoder vs. a pretrained visual encoder trained on different datasets and algorithms, such as R3M [8], VC1 [9], and MVP [7], or even only pretraining on ImageNet-1K [59], in Section 3.4.1.”

“VC-1 [187] did a comprehensive study on the effectiveness of vision-based pre-training on policy learning.”

“Prior works [65, 56, 57, 90] employ large-scale visual pre-training to develop embodied-aware pretrained representations, but these demonstrate limited generalization [58, 38, 42].”

“Representative examples include R3M [3] and VC-1 [53].”

“Visual representations are integrated into robot learning through pre-training [17, 18, 19], co-training [20, 21, 22, 23], or frozen encoders [24, 25, 26], all of which effectively support policy training.”

“A popular approach is to initialize policies from pre-trained vision encoders [33, 50, 53, 61, 74] or more recently, pre-trained vision-language models [7, 9, 11, 29].”

“Beyond directly acting as policy, pre-trained models can also guide policy learning from multiple aspects, such as providing good representations [45, 46, 47], providing reward signals [48, 49, 50], synthesizing goal images [18, 51], and predicting future sequences [17].”

“To further demonstrate the capability of PQ3D , we also transfer it to an embodied agent for object navigation using the ObjNav task from CortexBench [50] and instruction-tune it with a large language model (LLM) Vicuna-7B using the instruction-following dataset from [32].”

“However, most visual pre-trained networks are trained on ImageNet or human videos [24, 27, 52, 53], which exhibit a significant domain gap when compared to typical images from robot datasets, where the images frequently include robots or grippers.”

“We comprehensively evaluate Theia and baseline pre-trained models on the MuJoCo subset of Cor-texBench [20] to provide an overall assessment of pre-trained visual representations.”

“The prevalent approaches to building capable navigation agents include both end-to-end training [1, 6] and modular methods that leverage mapping [36, 37, 38] or off-the-shelf foundation models [39, 40, 41, 42, 43].”

“One intuitive approach is to utilize pre-trained representation models for images [24, 25, 26, 31, 38] to convert them into 1D representations and map these transformed observations to the action space either through a BC policy [55] or non-parametric nearest neighbour [34].”

“We leverage the off-the-shelf pre-trained visual encoders ( i.e. , VC-1 [56], CLIP [23], and DINO [66]) to imbue the training process with visual priors.”

“The perception data is provided by a RealSense d435 camera installed in the front of the robot.”

“…on training representations for images/videos in self-supervised learning manner (He et al., 2021; Xiao et al., 2022; Radosavovic et al., 2022; Majumdar et al., 2023; Radford et al., 2021; Nair et al., 2022; Ma et al., 2023; Oquab et al., 2023; Darcet et al., 2023; Kirillov et al., 2023;…”

“BLIP2 [48] offers a universal and efficient pre-training strategy, guiding visual language pre-training from ready-made image encoders and frozen language models.”

“One common way of using data beyond robot interactions for efficient learning is to pre-train the visual representations which serve as backbones for the policy models [62, 63, 64, 65, 66] with passive human videos [23, 67] and image data [68].”

“Given a schedule { α t } Tt =1 with α t = 1 − β t , a sample x 0 drawn from the clean distribution, the forward noising process follows a closed form: A denoiser ε θ is trained to recognize the noise inside the noisy sample via (3) to recover the clean sample.”

“Some prior robot learning approaches have also investigated leveraging out-of-domain (human) data, primarily for learning visual representations [2, 3, 4] and robotic affordances [5, 6, 7, 8].”

“[45] have comprehensively studied the impact of visual backbones on embodied performance.”

“In addition, we adopt the evaluation methodology from VC-1 (Majumdar et al., 2023) encompassing two key metrics: Mean Success and Mean Rank .”

“Masked auto-encoding approaches like MVP [3] and VC-1 [4] adapt MAE transformers [15] to human clips.”

“These methods have been extended to decision-making through effective downstream policy learning [7–11 , 6].”

“Inspired by successes in computer vision [1] and natural language processing [70], the robot learning community has recently focused on pretraining representations on large video datasets like Ego4D [49] and fine-tuning these representations on downstream tasks [4–6 , 8].”

“Apart from the CLIP-ViT pretrained encodings as we adopted in our main results, two other initialization of weights were considered: one sourced from the Visual Cortex 1 (VC-1) (Majumdar et al., 2023b), recognized for being a leading pretrained C.4 R ANK S IZE A BLATION S TUDY In order to understand the impact of rank…”

“VC-1 [223] 2024 Studies pre-training data scale using 4,000+ hours of video with MAE-trained transformers.”

“Much prior work has studied improving generalization in robot learning, such as by using pre-trained visual representations [36, 33, 30, 39, 11, 31, 20], or pre-training on diverse robot data [35, 34].”

“Numerous authors have suggested pre-training base models on egocentric/embodied data [57, 58] or grounded spatial reasoning tasks [59, 60], as the resulting representations might be more conducive to learning control.”

“We also evaluate pre-trained weights downloaded from two SOTA baselines (VC-1 [15], MVP [22]).”

“In contrast, robot equipped with CKR-ConceptNet starts an incorrect trajectories at step 0 and stops at an incorrect point at step n .”

“We compare various baseline methods, such as robotic 2D representation meth-ods [52, 54, 61], 3D representation methods [58, 59, 96], and 3D imitation learning policies [23, 24, 36].”

“We are also starting to see large models and representations custom trained for robotics [1], [28], [29], [30].”

“Majumdar et al. (2023) benchmarks the performance of pre-trained visual representations on a handful of manipulation environments, but they focus on in-domain performance and also investigate navigation environments.”

“One potential issue is that roboticists often use the same exact pre-training methods from the vision community, but merely apply them to a different data mix (e.g., VC-1 [56] applies MAE [39] to Ego4D [32]).”

“The increased performance of RT-2 compared to RT-1 and other adjusted baseline models (such as VC-1 (Majumdar et al. 2023), R3M (Nair et al. 2022), MOO (Stone et al. 2023)) is attributed to the vision-language backbone combining co-finetuning the pre-trained model jointly on robotics and web data, so that the model con…”

“Recent work has explored general-2 purpose visual pre-training using in-the-wild datasets [69, 36, 77, 53, 86, 49, 45] to accelerate policy learning in model-free manners.”

“Recent works have shifted towards learning continuous progress representations directly from large-scale video data [65, 66, 71].”

“Some approaches [2, 62, 63, 64, 68, 77, 87, 111] focus on visual representation learning from human videos, while others [4, 10, 29, 75, 103, 108] propose pre-training the policy backbone with auxiliary video or latent prediction objectives.”

“To compare different pre-trained visual representations, benchmarks are established by CORTEXBENCH (Majumdar et al. (2023)) and EmbCLIP (Khandelwal et al. (2022)) to assess which model could provide a better ”artificial visual cortex” for manipulation tasks.”

“Out of Domain Representation Learning for Control: An alternative set of works have emphasized the use of in-the-wild visual datasets for representation learning (Parisi et al., 2022; Shridhar et al., 2022; Khandelwal et al., 2022; Yen-Chen et al., 2020; Majumdar et al., 2023).”

“Indeed, recent work demonstrates that the human ego view is valuable for robot learning Bharadhwaj et al. (2023); Bahl et al. (2022); Majumdar et al. (2023); Nair et al. (2022); Mandikal and Grauman (2021).”

“Inspired by computer vision, recent work has applied methods such as contrastive learning (Nair et al., 2022; Yang et al., 2023) and masked autoen-coders (Radosavovic et al., 2023; Majumdar et al., 2023; Karamcheti et al., 2023; Yang et al., 2024b) to embodied AI.”

“…require strong assumptions about tasks, datasets, or robotic platforms [3,10,11,16,17,26,29,34,39– 41 , In order to decompose any long-horizon task using vision, general knowledge about visual task progression that can discern embedded subtasks in long, unsegmented task videos must be acquired.”

“Methods like VC-1 and MVP have prolonged training times due to their use of MAE (Feichtenhofer et al., 2022).”

“Some approaches [78, 85, 71, 89, 59] focus on the enhancement of individual components, such as pretrained visual representations, dynamics learning, and reasoning.”

“The robot planned a global path using A ∗ and followed this path using DWA [30] as the local planner.”

“Researchers propose to learn representations from human videos to assist in task execution [36, 60, 65].”

“…and stands as the state-of-the-art (SOTA) visual representation specifically designed for manipulation tasks; (ii) VC-1-Diffusion utilizes VC-1 [58] encoder (ViT-L) to extract image representations, which is also trained on large-scale egocentric videos [27] and ImageNet [71] using Masked…”

“3) Methods: We employ VINN [26] and its variants that alter the visual representations [22, 23, 29] as non-parametric methods.”

“In parallel, several studies [13, 24, 33, 36, 43–45, 49, 51, 67] have proposed methods to learn a generalizable representation from large-scale human datasets, such as Ego4D [21] and Kinetics [8].”

“Alternatively, some literature [30, 31, 33, 41, 45] proposes decoupling perception and control in the policy by using pre-trained 2D visual representations for manipulations, but they offer limited advantages over VFMs in generalization evaluations [3].”

“Many works focus on single-task settings (Nair et al. 2022a; Parisi et al. 2022; Sharma et al. 2023; Hansen et al. 2022), or simulated robot environments (Hansen et al. 2022; Majumdar et al. 2023).”

“There is a line of works that try to answer these questions through multi-view object consistency [7], spatial visual question answering [27, 122], visual perspective taking [54] and robot learning [57, 79, 119].”

“To explain our choice of CLIP [56] feature as a feature encoder, we additionally trained two versions of our method with image features produced by DINOv2 [50] and VC-1 [47] feature encoders.”

“, the benchmark tasks in [40]) to further understand the trade-offs between KODex and other imitation learning baselines.”

“[45] have comprehensively studied the impact of visual backbones on embodied performance.”

“These experiments thus recall the standard setting in which pre-trained visual representations are evaluated (Nair et al., 2022; Ma et al., 2023; Majumdar et al., 2023).”

“While recent work has leveraged pretraining on large-scale video datasets to create reusable vision representations for robot learning [1]–[3], there has been little focus on large-scale pretraining for other modalities such as tactile sensing.”

“Notably, MAE observes a substantial 25.5% gain on the MoveCan task.”

“We compared our method with three previous state-of-the-art approaches on the quadruped robot VLA task: QUART [24], VC-1 [36], and CLIP [35].”

“For some capabilities, such as visual navigation, simulation benchmark performance has positively correlated with real-world metrics (Kadian et al., 2020; Silwal et al., 2024) especially when utilizing encoders pretrained on real data - such as VC-1 (Majumdar et al., 2023) - in both sim and real experiments.”

“Several works leverage pre-trained visual representations to initialize perception encoders or extract latent image states [43, 44, 45, 46], while others use pre-trained LLMs or VLMs for linguistic instruction encoding [47, 48, 49].”

“…al., 2024) that primarily focused on 2D images, static point cloud or language-based spatial understanding, human comprehension of the world is grounded in continuous visual perception (Zhang et al., 2024; Liao et al., 2024; Majumdar et al., 2023), akin to embodied cognition through video streams.”

“Prior work has explored various strategies to extract knowledge from human videos, such as learning visual representations [36–38] and deriving reward functions [39–42].”

“Specifically, the robotics community has trended towards learning representations us-ing state-of-the-art self-supervised vision algorithms on diverse interaction datasets [8, 17, 39], and finetune the network on robotics tasks [9, 28–30, 33, 37, 42].”

“Previous robotic MAE methods typically use low-level RGB information [46, 54, 56] as the reconstruction target.”

“Recently, inspired by the success of transfer learning in CV, many works [69, 73, 65, 58, 59, 19] have explored developing a pre-trained visual representation (PVR) using large-scale out-of-domain data for various robotic motor control tasks.”