The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates
Open accessAbstract
Reinforcement learning holds the promise of enabling autonomous robots to learn large repertoires of behavioral skills with minimal human intervention. However, robotic applications of reinforcement learning often compromise the autonomy of the learning process in favor of achieving training times that are practical for real physical systems. This typically involves introducing hand-engineered policy representations and human-supplied demonstrations. Deep reinforcement learning alleviates this limitation by training general-purpose neural network policies, but applications of direct deep reinforcement learning algorithms have so far been restricted to simulated settings and relatively simple tasks, due to their apparent high sample complexity. In this paper, we demonstrate that a recent deep reinforcement learning algorithm based on off-policy training of deep Q-functions can scale to complex 3D manipulation tasks and can learn deep neural network policies efficiently enough to train on real physical robots. We demonstrate that the training times can be further reduced by parallelizing the algorithm across multiple robots which pool their policy updates asynchronously. Our experimental evaluation shows that our method can learn a variety of 3D manipulation skills in simulation and a complex door opening skill on real robots without any prior demonstrations or manually designed representations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Self-citationWatkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Self-citationReinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160
Learning and generalization of motor skills by learning from demonstration
Pastor, P., Hoffmann, H., Asfour, T., & Schaal, S. (2009) · 2009 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2009.5152385 · Cited by 702
Learning force control policies for compliant manipulation
Kalakrishnan, M., Righetti, L., Pastor, P., & Schaal, S. (2011) · 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2011.6095096 · Cited by 161
Research works printed but not linked: 34
- 10.1109/ROBOT.2009.5152577 · DOI · Paper page
- 10.1109/ICRA.2013.6631180 · DOI · Paper page
- kuffner, 2010, Cloud-enabled humanoid robots, IEEE-RAS International Conference on Humanoid Robotics
- 10.1177/02783640022067878 · DOI
- 10.1007/s10994-011-5235-x · DOI · Paper page
- 10.1109/IROS.2012.6386109 · DOI · Paper page
- 10.1109/ICSMC.2004.1400771 · doi:10.1109/icsmc.2004.1400771
- yahya, 2016, Col-lective robot reinforcement learning with distributed asynchronous guided policy search, arXiv preprint arXiv 1610 01292
- 10.1109/TASE.2014.2376492 · DOI · Paper page
- tedrake, 0, Learning to walk in 20 minutes
- gu, 2016, Continuous deep q-learning with model-based acceleration, ICML
- deisenroth, 2013, A survey on policy search for robotics, Foundations and Trends in Robotics, 2, 1, 10.1561/2300000021 · DOI · Paper page
- 10.1016/0005-1098(92)90053-I · DOI · Paper page
- riedmiller, 2005, Neural fitted q in experiences with a data efficient neural reinforcement learning method, European Conference on Machine Learning, 317
- hafner, 2007, Neural reinforcement learning controllers for a real robot application, ICRA
- riedmiller, 2012, Autonomous reinforcement learning on raw visual input data in a real world application, International Joint Conference on Neural Networks
- 10.1145/2463372.2463509 · DOI · Paper page
- schulman, 2015, Trust region policy optimization, ICML
- levine, 2016, End-to-end training of deep visuomotor policies, Journal of Machine Learning Resaerch, 17
- 10.1038/nature14236 · DOI · Paper page
- theodorou, 2010, Reinforcement learning of motor skills in high dimensions, ICRA
- 10.1145/2463372.2463509 · DOI · Paper page
- 10.1016/j.neunet.2008.02.003 · DOI · Paper page
- peters, 2010, Relative entropy policy search, AAAI Conference on Artificial Intelligence, 10.1609/aaai.v24i1.7727 · DOI · Paper page
- abbeel, 2006, An application of reinforcement learning to aerobatic helicopter flight, Advances in Neural Information Processing Systems (NIPS)
- 10.1177/0278364907084980 · DOI · Paper page
- 10.1109/ROBOT.2004.1307456 · DOI · Paper page
- deisenroth, 2011, PILCO: a model-based and data-efficient approach to policy search, ICML
- lioutikov, 2014, Sample-based information-theoretic stochastic optimal control, International Conference on Robotics and Automation
- 10.1109/ICRA.2015.7139645 · DOI
- rusu, 2016, Policy distillation, ICLRE
- 10.1007/BF00992696 · DOI · Paper page
- deisenroth, 2013, A survey on policy search for robotics, Foundations and Trends in Robotics, 2, 1, 10.1561/2300000021 · DOI · Paper page
- sutton, 1999, Policy gradient methods for reinforcement learning with function approximation, Advances in Neural Information Processing Systems (NIPS)
Cited by
Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594
Self-citationSoft Actor-Critic Algorithms and Applications
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018) · arXiv · doi:10.48550/arxiv.1812.05905 · Cited by 1,994
Self-citationLearning dexterous in-hand manipulation
Andrychowicz, O. M., Baker, B., Chociej, M., Józefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., et al. (2020) · The International Journal of Robotics Research · doi:10.1177/0278364919887447 · Cited by 1,643
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139
Transfer Learning in Deep Reinforcement Learning: A Survey
Zhu, Z., Lin, K., Jain, A. K., & Zhou, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292075 · Cited by 861
Metadata
- Authors
- Gu, S., Holly, E., Lillicrap, T., & Levine, S.
- Venue
- 2017 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2017-05
- Pages
- 3389-3396
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2575705757
- Cited by
- 1,424
- References
- 8
- FWCI
- 70.76
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. https://app.cochat.ai/papers/10.1109/icra.2017.7989385
- The paper
- Gu, S., Holly, E., Lillicrap, T., & Levine, S. (2017). Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. 2017 IEEE International Conference on Robotics and Automation (ICRA). https://doi.org/10.1109/icra.2017.7989385
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
