The DOI matches its Crossref record; 184 of the 193 references it cites were found in a registry or an index.
How to train your robot with deep reinforcement learning: lessons we have learned
Open accessAbstract
Deep reinforcement learning (RL) has emerged as a promising approach for autonomously acquiring complex behaviors from low-level sensor observations. Although a large portion of deep RL research has focused on applications in video games and simulated control, which does not connect with the constraints of learning in real environments, deep RL has also demonstrated promise in enabling physical robots to learn complex skills in the real world. At the same time, real-world robotics provides an appealing domain for evaluating such algorithms, as it connects directly to how humans learn: as an embodied agent in the real world. Learning to perceive and move in the real world presents numerous challenges, some of which are easier to address than others, and some of which are often not considered in RL research that focuses only on simulated domains. In this review article, we present a number of case studies involving robotic deep RL. Building off of these case studies, we discuss commonly perceived challenges in deep RL and how they have been addressed in these works. We also provide an overview of other outstanding challenges, many of which are unique to the real-world robotics setting and are not often the focus of mainstream RL research. Our goal is to provide a resource both for roboticists and machine learning researchers who are interested in furthering the progress of deep RL in the real world.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“POMDPs more accurately describe real-world applications of robotic systems [7] and are a major tool for training modern DRL and DIL end-to-end policy controllers.”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First, DRL is data-hungry and trained through trial and error which requires human involvement for constantly resetting the environment and preventing unsafe actions [7].”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“(4) high-dimensional/continuous state settings exacerbate the challenge of exploration and further affect data efficiency.”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We can also utilise demonstration data to further reduce the distribution shift, as it causes the model to learn the important part of the state space [7].”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [7]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In recent years, data-driven approaches with deep learning (DL) [6], such as deep reinforcement learning (DRL) [7] and deep imitation learning (DIL) [8], have demonstrated many advantages in performing task-level and skill-level control problems (Section 3.3).”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“(3) DRL algorithms often come with a set of hyper-parameters, and the algorithms are often sensitive to these hyper-parameters and the application domain”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“(2) data efficiency (both sampling and learning efficiency) is a major concern for such complex settings”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“MBRL is generally more data-efficient than MFRL algorithms and it generalises better with large and diverse data [7].”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [7]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“(1) theoretically, we have no guarantee of convergence of the DRL algorithms”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [7]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,268
Labaca-Castro, R. (2023) · Springer eBooks · doi:10.1007/978-3-658-40442-0_9 · Cited by 21,119
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 20,507
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 16,043
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881
(2017) · Cited by 11,625
Research works printed but not linked: 61
- Achiam J, 2017, International Conference on Machine Learning
- Agarwal R, 2020, International Conference on Machine Learning
- Andrychowicz M, 2017, Advances in Neural Information Processing Systems, 5048
- Burda Y, 2019, International Conference on Learning Representations
- Cabi S, 2019, arXiv preprint arXiv:1909.12200 · arXiv
- Chebotar Y, 2017, International Conference on Machine Learning, 703
- Chen Z, 2018, International Conference on Machine Learning
- Clavera I, 2019, International Conference on Learning Representations
- Deisenroth M, 2011, International Conference on Machine Learning, 465
- Deisenroth MP, 2013, Foundations and Trends in Robotics, 2, 1
- Devlin J, 2019, NAACL-HLT
- Duan Y, 2017, Advances in Neural Information Processing Systems, 1087
- Eysenbach B, 2018, International Conference on Learning Representations
- Finn C, 2017, International Conference on Machine Learning
- Finn C, 2016, International Conference on Machine Learning, 49
- Finn C, 2017, Proceedings of Machine Learning Research, 78, 357
- Fox R, 2016, Conference on Uncertainty in Artificial Intelligence
- Fu J, 2017, Advances in Neural Information Processing Systems, 2577
- Fujimoto S, 2019, International Conference on Machine Learning
- Fujimoto S, 2018, International Conference on Machine Learning
- Garofolo JS, 1993, NASA STI/Recon Technical Report, 93
- Goodfellow I, 2014, Advances in Neural Information Processing Systems, 2672
- Gregor K, 2017, International Conference on Learning Representations, Workshop Track Proceedings
- Gu S, 2016, International Conference on Machine Learning, 2829
- Haarnoja T, 2017, International Conference on Machine Learning, 1352
- Haarnoja T, 2018, International Conference on Machine Learning
- Hausman K, 2017, AAAI Symposium on Interactive Multi-Sensory Object Perception for Embodied Agents
- Iscen A, 2018, Conference on Robot Learning
- Jabri A, 2019, Advances in Neural Information Processing Systems, 10519
- James S, 2018, Conference on Robot Learning
- James S, 2017, Conference on Robot Learning
- Kalashnikov D, 2018, Conference on Robot Learning, Proceedings of Machine Learning Research
- Khadka S, 2019, International Conference on Machine Learning
- Kroemer O, 2019, CoRR abs/1907.03146 · arXiv
- Kumar A, 2019, Advances in Neural Information Processing Systems, 32
- Kurutach T, 2018, International Conference on Learning Representations
- Levine S, 2014, Advances in Neural Information Processing Systems, 1071
- Levine S, 2016, The Journal of Machine Learning Research, 17, 1334
- Levine S, 2013, International Conference on Machine Learning
- Mania H, 2018, Advances in Neural Information Processing Systems
- Montgomery WH, 2016, Advances in Neural Information Processing Systems, 4008
- Nagabandi A, 2020, Conference on Robot Learning
- Osband I, 2016, Advances in Neural Information Processing Systems, 4026
- Parisotto E, 2016, CoRR
- Paszke A, 2017, Advances in Neural Information Processing Systems Workshop on Autodiff
- Pinto L, 2017, International Conference on Machine Learning
- Rakelly K, 2019, International Conference on Machine Learning
- Riedmiller M, 2018, International Conference on Machine Learning
- Schaul T, 2019, Multidisciplinary Conference on Reinforcement Learning and Decision Making
- Schoettler G, 2019, International Conference on Intelligent Robots and Systems
- Schulman J, 2015, International Conference on Machine Learning
- Sener O, 2018, Advances in Neural Information Processing Systems, 527
- Tedrake R, 2015, Workshop on Adaptive and Learning Systems
- Viereck U, 2017, Conference on Robot Learning
- Wu YH, 2019, International Conference on Machine Learning, Proceedings of Machine Learning Research
- Xiao T, 2020, International Conference on Learning Representations
- Xie A, 2019, arXiv preprint arXiv:1904.05538 · arXiv
- Xie A, 2018, Proceedings of Machine Learning Research, 87, 40
- Yang Y, 2019, AAMAS
- Yang Y, 2020, Conference on Robot Learning
- Yen-Chen L, 2020, Conference on Robot Learning
Cited by
Sorted by citations
Manakitsa, N., Maraslidis, G. S., Moysis, L., & Fragulis, G. F. (2024) · Technologies · doi:10.3390/technologies12020015 · Cited by 355
Explainable Reinforcement Learning: A Survey and Comparative Review
Milani, S., Topin, N., Veloso, M., & Fang, F. (2023) · ACM Computing Surveys · doi:10.1145/3616864 · Cited by 347
Robot learning towards smart robotic manufacturing: A review
Liu, Z., Liu, Q., Xu, W., Wang, L., & Zhou, Z. (2022) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2022.102360 · Cited by 251
Recent advancements of robotics in construction
Xiao, B., Chen, C., & Yin, X. (2022) · Automation in Construction · doi:10.1016/j.autcon.2022.104591 · Cited by 204
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes
Tang, C., Abbatematteo, B., Hu, J., Chandra, R., Martín-Martín, R., & Stone, P. (2024) · Annual Review of Control Robotics and Autonomous Systems · doi:10.1146/annurev-control-030323-022510 · Cited by 276
Metadata
- Authors
- Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S.
- Affiliations
- Google (United States) (US)
- Everyday Robots, X, The Moonshot Factory, Mountain View, CA, USA
- Corresponding author
- Ibarz, Julian
- Venue
- The International Journal of Robotics Research
- Publisher
- SAGE Publications
- Published
- 2021-01-31
- Volume
- 40
- Issue
- 4-5
- Pages
- 698-721
- ISSN
- 0278-3649 (ISSN-L)
- 1741-3176
- OpenAlex
- W3126321819
- Cited by
- 570
- References
- 193
- Citations by year
- 2026: 63 · 2025: 123 · 2024: 145 · 2023: 112 · 2022: 89 · 2021: 38
- FWCI
- 33.92
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- The International Journal of Robotics Research (publishedVersion, cc-by-nc)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- deep reinforcement learning, robot learning, embodied agents, physical robots, robot skill learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
