Solving Rubik's Cube with a Robot Hand
Open accessAbstract
We demonstrate that models trained only in simulation can be used to solve a manipulation problem of unprecedented complexity on a real robot. This is made possible by two key components: a novel algorithm, which we call automatic domain randomization (ADR) and a robot platform built for machine learning. ADR automatically generates a distribution over randomized environments of ever-increasing difficulty. Control policies and vision state estimators trained with ADR exhibit vastly improved sim2real transfer. For control policies, memory-augmented models trained on an ADR-generated distribution of environments show clear signs of emergent meta-learning at test time. The combination of ADR with our custom robot platform allows us to solve a Rubik's cube with a humanoid robot hand, which involves both control and state estimation problems. Videos summarizing our results are available: https://openai.com/blog/solving-rubiks-cube/
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Challenges of training on synthetic data can be mitigated to some extent by using training data with greater diversity.
“These challenges can be mitigated to some extent by using training data with greater diversity (Fan et al., 2023; OpenAI et al., 2019)”
From Will we run out of data? Limits of LLM scaling based on human-generated data (2022) · cites this paper as OpenAI et al., 2019
In ADR, distributions parametrized by phi allow to randomly select parameters, and phi changes during learning procedure, unlike other DR approaches where distributions are parametrized with fixed phi.
“What changes from standard DR idea are ( 0 ) and ( ( k ) ) distributions (parametrized by ) that allow to randomly select ( k ) and 0 . Indeed, while in other DR approaches these distributions are parametrized with xed , in ADR changes during learning procedure.”
From Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning (2021) · cites this paper as [57]
In [57], Automatic Domain Randomisation (ADR) is proposed in order to transfer a policy learnt in simulation on the real system, framing it in a manipulation task of Rubik's cube with a robotic hand.
“In a quite recent work [57], Automatic Domain Randomisation (ADR) is proposed in order to transfer a policy learnt in simulation on the real system, framing it in a manipulation task of Rubik's cube with a robotic hand.”
From Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning (2021) · cites this paper as [57]
In [57], a sim-to-real transfer is performed, with worse results.
“In addition, a sim-to-real transfer is performed, with worse results.”
From Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning (2021) · cites this paper as [57]
The developed controller in [57] has been firstly tested in environments in which distributions were manually tuned, achieving good results.
“The developed controller has been rstly tested in environments in which distributions were manually tuned, achieving good results.”
From Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning (2021) · cites this paper as [57]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
Soft Actor-Critic Algorithms and Applications
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018) · arXiv · doi:10.48550/arxiv.1812.05905 · Cited by 1,994 · see citing works
Learning to reinforcement learn
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., & Botvinick, M. (2016) · arXiv · doi:10.48550/arxiv.1611.05763 · Cited by 579 · see citing works
Exploration by Random Network Distillation
Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018) · arXiv · doi:10.48550/arxiv.1810.12894 · Cited by 527 · see citing works
Robust adversarial reinforcement learning
Pinto, L., Davidson, J., Sukthankar, R., & Gupta, A. (2017) · arXiv · doi:10.48550/arxiv.1703.02702 · Cited by 507 · cited 2× · see citing works
“[83] propose to use adversarial training to obtain more robust policies and show that it also helps with transfer to physical robots [82].”
BackgroundMethodSim-to-Real Robot Learning from Pixels with Progressive Nets
Rusu, A. A., Vecerik, M., Rothörl, T., Heess, N., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1610.04286 · Cited by 385 · see citing works
Preparing for the unknown: Learning a universal policy with online system identification
Yu, W., Tan, J., Karen Liu, C., & Turk, G. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.048 · Cited by 237 · see citing works
Clune, J. (2019) · arXiv · doi:10.48550/arxiv.1905.10985 · Cited by 82 · see citing works
POET: open-ended coevolution of environments and their optimized solutions
Wang, R., Lehman, J., Clune, J., & Stanley, K. O. (2019) · Proceedings of the Genetic and Evolutionary Computation Conference · doi:10.1145/3321707.3321799 · Cited by 61 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224 · see citing works
Learning quadrupedal locomotion over challenging terrain
Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., & Hutter, M. (2020) · Science Robotics · doi:10.1126/scirobotics.abc5986 · Cited by 1,156 · see citing works
Dota 2 with Large Scale Deep Reinforcement Learning
OpenAI, :, Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.06680 · Cited by 1,065 · see citing works
Self-citationDeep Learning applications for COVID-19
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal of Big Data · doi:10.1186/s40537-020-00392-9 · Cited by 350 · see citing works
Embodied intelligence via learning and evolution
Gupta, A., Savarese, S., Ganguli, S., & Fei-Fei, L. (2021) · Nature Communications · doi:10.1038/s41467-021-25874-z · Cited by 352 · see citing works
Deep reinforcement learning and its neuroscientific implications
Botvinick, M., Wang, J. X., Dabney, W., Miller, K. J., & Kurth-Nelson, Z. (2020) · Neuron · doi:10.1016/j.neuron.2020.06.014 · Cited by 269 · see citing works
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518 · see citing works
Machine Learning for Soft Robotic Sensing and Control
Chin, K., Hellebrekers, T., & Majidi, C. (2020) · Advanced Intelligent Systems · doi:10.1002/aisy.201900171 · Cited by 215 · see citing works
Salvato, E., Fenu, G., Medvet, E., & Pellegrino, F. A. (2021) · IEEE Access · doi:10.1109/access.2021.3126658 · Cited by 200 · see citing works
“lation [54] PPO Trotting and galloping of quadruped [56] A3C Marble maze game with robotic arm [57] PPO Rubik’s cube with robotic hand [46] QT-Opt Rvision-based control task of a robotic arm for grasping [61] PPO Climb and descend stairs with bipedal robot [41] A3C Wheeled mobile platform navigation”
BackgroundMethodInfluentialA Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193 · see citing works
“…Learning (RL) has the potential to be used in a wide range of applications from autonomous vehicles (Filos et al., 2020) and algorithm control (Biedenkapp et al., 2020) to robotics (OpenAI et al., 2019a), but to fulfil this potential we need RL algorithms that can be used in the real world.”
BackgroundMethodSim2Real in Robotics and Automation: Applications and Challenges
Hofer, S., Bekris, K., Handa, A., Gamboa, J. C., Mozifian, M., Golemo, F., Atkeson, C., Fox, D., Goldberg, K., Leonard, J., et al. (2021) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2021.3064065 · Cited by 154 · see citing works
Self-citationAugmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks
Nasiriany, S., Liu, H., & Zhu, Y. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9812140 · Cited by 91 · see citing works
Will we run out of data? Limits of LLM scaling based on human-generated data
Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2022) · arXiv · doi:10.48550/arxiv.2211.04325 · Cited by 80 · see citing works
“These challenges can be mitigated to some extent by using training data with greater diversity (Fan et al., 2023; OpenAI et al., 2019), or by training on a mixture of human-generated and synthetic data (Gunasekar et al., 2023; Shumailov et al., 2023; Gerstgrasser et al., 2024; Alemohammad et al.,…”
BackgroundThe overfitted brain: Dreams evolved to assist generalization
Hoel, E. (2021) · Patterns · doi:10.1016/j.patter.2021.100244 · Cited by 70 · see citing works
Understanding World or Predicting Future? A Comprehensive Survey of World Models
Ding, J., Zhang, Y., Shang, Y., Zhang, Y., Zong, Z., Feng, J., Yuan, Y., Su, H., Li, N., Sukiennik, N., et al. (2026) · ACM Computing Surveys · doi:10.1145/3746449 · Cited by 58 · see citing works
DayDreamer: World Models for Physical Robot Learning
Wu, P., Escontrela, A., Hafner, D., Goldberg, K., & Abbeel, P. (2022) · arXiv · doi:10.48550/arxiv.2206.14176 · Cited by 76 · see citing works
Chaffre, T., Moras, J., Chan-Hon-Tong, A., & Marzat, J. (2020) · Proceedings of the 17th International Conference on Informatics in Control, Automation and Robotics · doi:10.5220/0009821603140323 · Cited by 38 · see citing works
ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
Mu, T., Ling, Z., Xiang, F., Yang, D., Li, X., Tao, S., Huang, Z., Jia, Z., & Su, H. (2021) · arXiv · doi:10.48550/arxiv.2107.14483 · Cited by 34 · see citing works
The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels
Stone, A., Ramirez, O., Konolige, K., & Jonschkowski, R. (2021) · arXiv · doi:10.48550/arxiv.2101.02722 · Cited by 31 · see citing works
Temporally resolved neural dynamics underlying handwriting
Pei, L., Longcamp, M., Leung, F. K.-S., & Ouyang, G. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118578 · Cited by 23 · see citing works
SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning
Lv, J., Yu, Q., Shao, L., Liu, W., Xu, W., & Lu, C. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9811859 · Cited by 22 · see citing works
Replay-Guided Adversarial Environment Design
Jiang, M., Dennis, M., Parker-Holder, J., Foerster, J., Grefenstette, E., & Rocktäschel, T. (2021) · arXiv · doi:10.48550/arxiv.2110.02439 · Cited by 21 · see citing works
Human-Timescale Adaptation in an Open-Ended Task Space
Adaptive Agent Team, Bauer, J., Baumli, K., Baveja, S., Behbahani, F., Bhoopchand, A., Bradley-Schmieg, N., Chang, M., Clay, N., Collister, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.07608 · Cited by 19 · see citing works
“Recent works have demonstrated the effectiveness of agent-environment co-adaptation with a distribution of tasks (Wang et al., 2019; Akkaya et al., 2019; Wang et al., 2020a; Parker-Holder et al., 2022).”
BackgroundSelf-citationLearning few-shot imitation as cultural transmission
Bhoopchand, A., Brownfield, B., Collister, A., Dal Lago, A., Edwards, A., Everett, R., Fréchette, A., Oliveira, Y. G., Hughes, E., Mathewson, K. W., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-42875-2 · Cited by 14 · see citing works
Self-citationMeasuring Visual Generalization in Continuous Control from Pixels
Grigsby, J., & Qi, Y. (2020) · arXiv · doi:10.48550/arxiv.2010.06740 · Cited by 10 · see citing works
FOCUS: object-centric world models for robotic manipulation
Ferraro, S., Mazzaglia, P., Verbelen, T., & Dhoedt, B. (2025) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2025.1585386 · Cited by 9 · see citing works
“In recent years, deep reinforcement learning (RL) has shown to be a promising approach for dealing with complex manipulation scenarios (Levine et al., 2016; OpenAI et al., 2019; Kalashnikov et al., 2018; Lu et al., 2021; Lee et al., 2021; Ferraro et al., 2022a).”
Design and evaluation of a global workspace agent embodied in a realistic multimodal environment
Dossa, R. F. J., Arulkumaran, K., Juliani, A., Sasai, S., & Kanai, R. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1352685 · Cited by 5 · see citing works
Generalization through Diversity: Improving Unsupervised Environment Design
Li, W., Varakantham, P., & Li, D. (2023) · arXiv · Cited by 3 · see citing works
“Deep Reinforcement Learning (DRL) has had many successes in challenging tasks (e.g., Atari [Mnih et al. , 2015], AlphaGO [Silver et al. , 2016], solving Rubik’s cube [Akkaya et al. , 2019], chip design [Mirhoseini et al. , 2021]) in the last decade.”
BackgroundGeneralization through Diversity: Improving Unsupervised Environment Design
Li, W., Varakantham, P., & Li, D. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/601 · Cited by 3 · see citing works
“Deep Reinforcement Learning (DRL) has had many successes in challenging tasks (e.g., Atari [Mnih et al. , 2015], AlphaGO [Silver et al. , 2016], solving Rubik’s cube [Akkaya et al. , 2019], chip design [Mirhoseini et al. , 2021]) in the last decade.”
BackgroundImproving Environment Novelty Quantification for Effective Unsupervised Environment Design
Teoh, J., Li, W., & Varakantham, P. (2025) · arXiv
“Although recent advancements in Deep Reinforcement Learning (DRL) have led to many successes, e.g., super-human performance in games [26, 9] and reliable control in robotics [2, 3], training generally-capable agents remains a significant challenge.”
Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration
Chenu, A., Serris, O., Sigaud, O., & Perrin-Gilbert, N. (2022) · arXiv · doi:10.48550/arxiv.2211.04786
“Deep reinforcement learning (DRL) offers an al-1 Sorbonne Université, CNRS, Institut des Systèmes Intelligents et de Robotique, ISIR F-75005 Paris, France chenu.isir@gmail.com ternative, learning control policies from reward signals alone [1, 3, 42].”
Improving Environment Novelty Quantification for Effective Unsupervised Environment Design
Teoh, J., Li, W., & Varakantham, P. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-4300
Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning
Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.10470 · Cited by 509 · see citing works
- Authors
- OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., Schneider, J., Tezak, N., Tworek, J., Welinder, P., Weng, L., Yuan, Q., Zaremba, W., & Zhang, L.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1910.07113
- OpenAlex
- W2981030070
- Cited by
- 602
- References
- 11
- Citations by year
- 2026: 5 · 2025: 17 · 2024: 64 · 2023: 105 · 2022: 90 · 2021: 173 · 2020: 142 · 2019: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- Rubik's Cube solving, sim-to-real transfer, meta-learning, humanoid robots, control policy learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
