Open X-Embodiment: Robotic Learning Datasets and RT-X Models

RT-2-X trains a 55B-parameter VLA policy on the Open X-Embodiment dataset and demonstrates state-of-the-art generalist manipulation policy performance.

“RT-2-X [1] trains a 55B-parameter VLA policy on the Open X-Embodiment dataset and demonstrates state-of-the-art generalist manipulation policy performance.”

Prior transformer-based policy designs typically encode input images with large ResNet-style encoders and fuse the resulting image features with a comparatively small transformer.

“Prior transformer-based policy designs typically encode input images with large ResNetstyle [32] encoders and fuse the resulting image features with a comparatively small transformer [10, 67, 81, 17, 98, 61, 83].”

The robots and tasks in open-source datasets typically have one or two cameras and use low-frequency control, between 2 and 10 Hz.

“The robots and tasks in these datasets typically have one or two cameras and use lowfrequency control, between 2 and 10 Hz.”

OpenVLA is a 7B-parameter open-source vision-language-action model trained on 970k robot demonstrations from the Open X-Embodiment dataset.

“we introduce OpenVLA, a 7B-parameter open-source VLA trained on a diverse collection of 970k real-world robot demonstrations.”

Even the largest robot manipulation datasets only have 100K to 1M examples.

“even the largest robot manipulation datasets [1, 11] only have 100K to 1M examples”

OpenVLA is trained on 970k trajectories, while RT-2-X is trained on 350k trajectories.

“we curated a much larger training dataset for OpenVLA with 970k trajectories (vs. 350k for RT-2-X)”

Octo is a transformer-based policy pretrained on 800k robot episodes from the Open X-Embodiment dataset.

“we introduce Octo, a large transformer-based policy trained on 800k trajectories from the Open X-Embodiment dataset, the largest robot manipulation dataset to date.”

Octo is trained on the largest robot manipulation dataset to date, 800k robot demonstrations from Open X-Embodiment.

“pretrained on the largest robot manipulation dataset to date: 800k robot demonstrations from the Open X-Embodiment dataset [67].”

The Open-X dataset contains approximately 1.5M robot episodes, of which 800k were curated for Octo training.

“The Open-X dataset contains approximately 1.5M robot episodes, of which we curate 800k for Octo training.”

The RT-X model used a more restricted subset of 350K episodes.

“We note that the RT-X model [67] used a more restricted subset of 350K episodes”

Vision-language-action and cross-embodiment datasets motivate action-conditioned models for physical systems.

“Vision-language-action and cross-embodiment datasets motivate action-conditioned models for physical systems [39-41].”

When using natural language to specify tasks, Octo outperforms RT-1-X across three different robot embodiments and setups.

“When using natural language to specify tasks, Octo outperforms RT-1-X [67], the current best openly available generalist robot policy across three different robot embodiments and setups.”

The pre-training mixture consists of 9.1% open-source datasets, including OXE, Bridge v2, and DROID.

“9 . 1% of the training mixture consists of open-source datasets, including OXE [10], Bridge v2 [52], and DROID [23].”

The open-source datasets cover a wide range of objects and environments.

“However, these datasets cover a wide range of objects and environments.”

The pre-training mixture consists of 10,000 hours of dexterous manipulation data from 7 different robot configurations and 68 tasks, in addition to large amounts of previously collected robot manipulation data from OXE, DROID, and Bridge.

“Our pre-training mixture consists of 10,000 hours of dexterous manipulation data from 7 different robot configurations and 68 tasks, in addition to large amounts of previously collected robot manipulation data from OXE [10], DROID [23], and Bridge [52].”

The pre-training mixture consists of a weighted combination of our own dexterous manipulation datasets collected on 7 different robot configurations for 68 different tasks, and the entire OXE dataset, which contains data from 22 robots.

“In our training framework, we first assemble a pre-training mixture consisting of a weighted combination of our own dexterous manipulation datasets (Section V-C), collected on 7 different robot configurations for 68 different tasks, and the entire OXE dataset [10], which contains data from 22 robots.”

Reproducing Internet-scale pretraining for robotics is still an open challenge.

“While reproducing this scale of pretraining for robotics is still an open challenge”

Octo is trained on the most diverse cross-embodied robot dataset to date, a mix of 25 datasets manually curated from the Open X-Embodiment dataset.

“Octo is trained on the most diverse crossembodied robot dataset to date, a mix of 25 datasets that we manually curated from the Open X-Embodiment dataset [67] (see Section III-B).”

OpenVLA outperforms RT-2-X by 16.5% absolute task success rate across 29 tasks and multiple robot embodiments.

“outperforming closed models such as RT-2-X (55B) by 16.5% in absolute task success rate across 29 tasks and multiple robot embodiments, with 7x fewer parameters.”

Existing VLAs are largely closed and inaccessible to the public.

“current models [1, 7, 17, 18] are closed , with limited visibility into model architecture, training procedures, and data mixture”