Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration 0
Abstract
Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train "generalist" X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. The project website is robotics-transformer-x.github.io.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Models such as RT-2, RT-X, and OpenVLA typically build upon language models or vision-language models that were pre-trained on text or multimodal data before being fine-tuned for robotic control.
“Models such as RT-2 (Zitkovich et al., 2023), RT-X (O'Neill et al., 2024), and OpenVLA (Kim et al., 2024) typically build upon language models or vision-language models that were pre-trained on text or multimodal data before being finetuned for robotic control.”
From The Vector Grounding Problem (2026) · cites this paper as O'Neill et al., 2024
Cites
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · see citing works
Film: Visual reasoning with a general conditioning layer
Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11671 · Cited by 2,490 · see citing works
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.01691 · Cited by 624 · see citing works
Self-citationPaLM-E: An Embodied Multimodal Language Model
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03378 · Cited by 591 · see citing works
Self-citationRT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15818 · Cited by 276 · cited 10× · see citing works
“To that end, our experiments will build on two recently proposed Transformer-based robotic policies: RT1 [8] and RT-2 [9].”
InfluentialSelf-citationCLIPort: What and Where Pathways for Robotic Manipulation
Shridhar, M., Manuelli, L., & Fox, D. (2021) · arXiv · doi:10.48550/arxiv.2109.12098 · Cited by 223 · see citing works
Affordances from Human Videos as a Versatile Representation for Robotics
Bahl, S., Mendonca, R., Chen, L., Jain, U., & Pathak, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01324 · Cited by 119 · see citing works
Self-citationInteractive Language: Talking to Robots in Real Time
Lynch, C., Wahid, A., Tompson, J., Ding, T., Betker, J., Baruch, R., Armstrong, T., & Florence, P. (2022) · arXiv · doi:10.48550/arxiv.2210.06407 · Cited by 113 · see citing works
Self-citationAVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos
Smith, L., Dhawan, N., Zhang, M., Abbeel, P., & Levine, S. (2020) · Robotics: Science and Systems XVI · doi:10.15607/rss.2020.xvi.024 · Cited by 102 · see citing works
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?
Majumdar, A., Yadav, K., Arnaud, S., Ma, Y. J., Chen, C., Silwal, S., Jain, A., Berges, V.-P., Abbeel, P., Malik, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18240 · Cited by 49 · see citing works
Self-citationMandlekar, A., Booher, J., Spero, M., Tung, A., Gupta, A., Zhu, Y., Garg, A., Savarese, S., & Fei-Fei, L. (2019) · arXiv · doi:10.48550/arxiv.1911.04052 · Cited by 48 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., & Xie, S. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.00994 · Cited by 120 · see citing works
Self-citationMultimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
Self-citationScaling Instructable Agents Across Many Simulated Worlds
SIMA Team, Raad, M. A., Ahuja, A., Barros, C., Besse, F., Bolt, A., Bolton, A., Brownfield, B., Buttimore, G., Cant, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.10179 · Cited by 9 · see citing works
“…has focused on environment-generality, including scaling robotic learning datasets across multiple tasks and embodiments (Brohan et al., 2022, 2023a; Stone et al., 2023; Padalkar et al., 2023)—thereby creating Vision-Language-Action (VLA) models (Brohan et al., 2023a), similar to the SIMA agent.”
Self-citationGemini Robotics: Bringing AI into the Physical World
Gemini Robotics Team, Abeyruwan, S., Ainslie, J., Alayrac, J.-B., Arenas, M. G., Armstrong, T., Balakrishna, A., Baruch, R., Bauza, M., Blokzijl, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.20020 · Cited by 8 · see citing works
Self-citationCoelho Mollo, D., & Millière, R. (2026) · Philosophy and the Mind Sciences · doi:10.33735/phimisci.2026.12307 · Cited by 48 · see citing works
GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
NVIDIA, :, Bjorck, J., Castañeda, F., Cherniadev, N., Da, X., Ding, R., Fan, L. "., Fang, Y., Fox, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.14734 · Cited by 10 · see citing works
Self-citationSmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics
Shukor, M., Aubakirova, D., Capuano, F., Kooijmans, P., Palma, S., Zouitine, A., Aractingi, M., Pascal, C., Russi, M., Marafioti, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.01844 · Cited by 3 · see citing works
Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks
Matthews, M., Beukman, M., Lu, C., & Foerster, J. (2024) · arXiv · doi:10.48550/arxiv.2410.23208 · Cited by 1 · see citing works
Self-citationCross-robot behavior adaptation through intention alignment
Chen, X., Gao, Y., Liu, H., Yang, F., Ghadirzadeh, A., Yang, J., Liang, B., Zhang, C., Lam, T. L., & Zhu, S.-C. (2026) · Science Robotics · doi:10.1126/scirobotics.adv2250
- Authors
- O’Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., Tung, A., Bewley, A., Herzog, A., Irpan, A., Khazatsky, A., Rai, A., Gupta, A., Wang, A., Singh, A., Garg, A., Kembhavi, A., Xie, A., Brohan, A., Raffin, A., Sharma, A., Yavary, A., Jain, A., Balakrishna, A., Wahid, A., Burgess-Limerick, B., Kim, B., Schölkopf, B., Wulfe, B., Ichter, B., Lu, C., Xu, C., Le, C., Finn, C., Wang, C., Xu, C., Chi, C., Huang, C., Chan, C., Agia, C., Pan, C., Fu, C., Devin, C., Xu, D., Morton, D., Driess, D., Chen, D., Pathak, D., Shah, D., Büchler, D., Jayaraman, D., Kalashnikov, D., Sadigh, D., Johns, E., Foster, E., Liu, F., Ceola, F., Xia, F., Zhao, F., Stulp, F., Zhou, G., Sukhatme, G. S., Salhotra, G., Yan, G., Feng, G., Schiavi, G., Berseth, G., Kahn, G., Wang, G., Su, H., Fang, H.-S., Shi, H., Bao, H., Ben Amor, H., Christensen, H. I., Furuta, H., Walke, H., Fang, H., Ha, H., Mordatch, I., Radosavovic, I., Leal, I., Liang, J., Abou-Chakra, J., Kim, J., Drake, J., Peters, J., Schneider, J., Hsu, J., Bohg, J., Bingham, J., Wu, J., Gao, J., Hu, J., Wu, J., Wu, J., Sun, J., Luo, J., Gu, J., Tan, J., Oh, J., Wu, J., Lu, J., Yang, J., Malik, J., Silvério, J., Hejna, J., Booher, J., Tompson, J., Yang, J., Salvador, J., Lim, J. J., Han, J., Wang, K., Rao, K., Pertsch, K., Hausman, K., Go, K., Gopalakrishnan, K., Goldberg, K., Byrne, K., Oslund, K., Kawaharazuka, K., Black, K., Lin, K., Zhang, K., Ehsani, K., Lekkala, K., Ellis, K., Rana, K., Srinivasan, K., Fang, K., Singh, K. P., Zeng, K.-H., Hatch, K., Hsu, K., Itti, L., Chen, L. Y., Pinto, L., Fei-Fei, L., Tan, L., Fan, L. J., Ott, L., Lee, L., Weihs, L., Chen, M., Lepert, M., Memmel, M., Tomizuka, M., Itkina, M., Castro, M. G., Spero, M., Du, M., Ahn, M., Yip, M. C., Zhang, M., Ding, M., Heo, M., Srirama, M. K., Sharma, M., Kim, M. J., Kanazawa, N., Hansen, N., Heess, N., Joshi, N. J., Suenderhauf, N., Liu, N., Di Palo, N., Shafiullah, N. M. M., Mees, O., Kroemer, O., Bastani, O., Sanketi, P. R., Miller, P. T., Yin, P., Wohlhart, P., Xu, P., Fagan, P. D., Mitrano, P., Sermanet, P., Abbeel, P., Sundaresan, P., Chen, Q., Vuong, Q., Rafailov, R., Tian, R., Doshi, R., Martín-Martín, R., Baijal, R., Scalise, R., Hendrix, R., Lin, R., Qian, R., Zhang, R., Mendonca, R., Shah, R., Hoque, R., Julian, R., Bustamante, S., Kirmani, S., Levine, S., Lin, S., Moore, S., Bahl, S., Dass, S., Sonawani, S., Song, S., Xu, S., Haldar, S., Karamcheti, S., Adebola, S., Guist, S., Nasiriany, S., Schaal, S., Welker, S., Tian, S., Ramamoorthy, S., Dasari, S., Belkhale, S., Park, S., Nair, S., Mirchandani, S., Osa, T., Gupta, T., Harada, T., Matsushima, T., Xiao, T., Kollar, T., Yu, T., Ding, T., Davchev, T., Zhao, T. Z., Armstrong, T., Darrell, T., Chung, T., Jain, V., Vanhoucke, V., Zhan, W., Zhou, W., Burgard, W., Chen, X., Wang, X., Zhu, X., Geng, X., Liu, X., Liangwei, X., Li, X., Lu, Y., Ma, Y. J., Kim, Y., Chebotar, Y., Zhou, Y., Zhu, Y., Wu, Y., Xu, Y., Wang, Y., Bisk, Y., Cho, Y., Lee, Y., Cui, Y., Cao, Y., Wu, Y.-H., Tang, Y., Zhu, Y., Zhang, Y., Jiang, Y., Li, Y., Li, Y., Iwasawa, Y., Matsuo, Y., Ma, Z., Xu, Z., Cui, Z. J., Zhang, Z., & Lin, Z.
- Affiliations
- University of California,Berkeley
- University of Edinburgh (GB)
- The University of Texas at Austin (US)
- University of Washington (US)
- Deutsches Zentrum für Luft- und Raumfahrt e. V. (DLR) (DE)
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Nvidia (United Kingdom) (GB)
- Stanford University (US)
- New York University (US)
- Georgia Institute of Technology (US)
- Allen Institute for Artificial Intelligence (US)
- University of California, Davis (US)
- Toyota Research Institute (US)
- Queensland University of Technology (AU)
- Korea Advanced Institute of Science and Technology (KR)
- Max Planck Society (DE)
- Shanghai Jiao Tong University (CN)
- Columbia University (US)
- University of Freiburg (DE)
- Carnegie Mellon University (US)
- University of Pennsylvania (US)
- Imperial College London (GB)
- Italian Institute of Technology (IT)
- IO-AI TECH
- University of Southern California (US)
- University of California San Diego (US)
- ETH Zurich (CH)
- Université de Montréal (CA)
- California Institute of Technology (US)
- Arizona State University
- The University of Tokyo (JP)
- Korea Advanced Institute of Science & Technology
- Technische Universität Darmstadt (DE)
- Google (United States) (US)
- Intrinsic LLC
- École Polytechnique Fédérale de Lausanne (CH)
- University of Illinois Urbana-Champaign (US)
- Imperial College London
- University of Michigan (US)
- Arizona State University (US)
- University of Technology Nuremberg (DE)
- ORCID
- Rehman, Abdul
- Gupta, Abhishek
- Padalkar, Abhishek
- Gupta, Agrim
- Mandlekar, Ajay
- Tung, Albert
- Bewley, Alex
- Wang, Andrew
- Garg, Animesh
- Kembhavi, Aniruddha
- Xie, Annie
- Raffin, Antonin
- Balakrishna, Ashwin
- Burgess-Limerick, Ben
- Kim, Beomjoon
- Schölkopf, Bernhard
- Wulfe, Blake
- Ichter, Brian
- Lu, Cewu
- Xu, Charles
- Finn, Chelsea
- Wang, Chen
- Xu, Chenfeng
- Chi, Cheng
- Huang, Chenguang
- Agia, Christopher
- Xu, Danfei
- Driess, Danny
- Pathak, Deepak
- Shah, Dhruv
- Büchler, Dieter
- Jayaraman, Dinesh
- Sadigh, Dorsa
- Johns, Edward
- Foster, Ethan
- Liu, Fangchen
- Ceola, Federico
- Xia, Fei
- Zhao, Feiyu
- Stulp, Freek
- Zhou, Gaoyue
- Sukhatme, Gaurav S.
- Salhotra, Gautam
- Yan, Ge
- Feng, Gilbert
- Berseth, Glen
- Kahn, Gregory
- Wang, Guanzhi
- Su, Hao
- Shi, Haochen
- Bao, Henghui
- Christensen, Henrik I
- Furuta, Hiroki
- Fang, Hongjie
- Radosavovic, Ilija
- Leal, Isabel
- Liang, Jacky
- Abou-Chakra, Jad
- Peters, Jan
- Hsu, Jasmine
- Bohg, Jeannette
- Gao, Jensen
- Wu, Jiajun
- Wu, Jialin
- Sun, Jiankai
- Luo, Jianlan
- Gu, Jiayuan
- Tan, Jie
- Oh, Jihoon
- Lu, Jingpei
- Yang, Jingyun
- Malik, Jitendra
- Silvério, João
- Hejna, Joey
- Salvador, Jordi
- Lim, Joseph J.
- Wang, Kaiyuan
- Hausman, Karol
- Go, Keegan
- Goldberg, Ken
- Kawaharazuka, Kento
- Lin, Kevin
- Zhang, Kevin
- Lekkala, Kiran
- Rana, Krishan
- Srinivasan, Krishnan
- Fang, Kuan
- Singh, Kunal Pratap
- Itti, Laurent
- Chen, Lawrence Yunliang
- Pinto, Lerrel
- Fei-Fei, Li
- Ott, Lionel
- Lee, Lisa
- Weihs, Luca
- Tomizuka, Masayoshi
- Itkina, Masha
- Castro, Mateo Guaman
- Yip, Michael C.
- Zhang, Mingtong
- Ding, Mingyu
- Heo, Minho
- Sharma, Mohit
- Kanazawa, Naoaki
- Hansen, Nicklas
- Heess, Nicolas
- Liu, Ning
- Shafiullah, Nur Muhammad Mahi
- Mees, Oier
- Kroemer, Oliver
- Bastani, Osbert
- Miller, Patrick Tree
- Wohlhart, Paul
- Xu, Peng
- Mitrano, Peter
- Abbeel, Pieter
- Chen, Qiuyu
- Vuong, Quan
- Tian, Ran
- Martín-Martín, Roberto
- Baijal, Rohan
- Scalise, Rosario
- Zhang, Ruohan
- Hoque, Ryan
- Levine, Sergey
- Lin, Shan
- Song, Shuran
- Haldar, Siddhant
- Karamcheti, Siddharth
- Adebola, Simeon
- Schaal, Stefan
- Tian, Stephen
- Ramamoorthy, Subramanian
- Dasari, Sudeep
- Belkhale, Suneel
- Park, Sungjae
- Nair, Suraj
- Osa, Takayuki
- Gupta, Tanmay
- Harada, Tatsuya
- Matsushima, Tatsuya
- Kollar, Thomas
- Davchev, Todor
- Darrell, Trevor
- Jain, Vidhi
- Vanhoucke, Vincent
- Zhan, Wei
- Zhou, Wenxuan
- Burgard, Wolfram
- Wang, Xiaolong
- Zhu, Xinghao
- Geng, Xinyang
- Liu, Xiyuan
- Li, Xuanlin
- Lu, Yao
- Kim, Yejin
- Zhu, Yifeng
- Wu, Yilin
- Xu, Ying
- Wang, Yixuan
- Bisk, Yonatan
- Cho, Yoonyoung
- Lee, Youngwoon
- Cui, Yuchen
- Cao, Yue
- Wu, Yueh-Hua
- Tang, Yujin
- Zhu, Yuke
- Zhang, Yunchu
- Jiang, Yunfan
- Li, Yunshuang
- Li, Yunzhu
- Iwasawa, Yusuke
- Matsuo, Yutaka
- Xu, Zhuo
- Zhang, Zichen
- Lin, Zipeng
- Venue
- 2024 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2024-05-13
- Pages
- 6892-6903
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W4401415287
- Cited by
- 210
- References
- 12
- Citations by year
- 2026: 110 · 2025: 100
- FWCI
- 131.01
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- elib (German Aerospace Center) (submittedVersion)
- FreiDok plus (Universitätsbibliothek Freiburg) (submittedVersion)
- KAIST Institutional Repository (KAIST) (submittedVersion)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- robot learning, robotic manipulation, transfer learning, robot datasets
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
