The DOI matches its Crossref record; 40 of the 44 references it cites were found in a registry or an index.
Overcoming Exploration in Reinforcement Learning with Demonstrations
Open accessAbstract
Exploration in environments with sparse rewards has been a persistent problem in reinforcement learning (RL). Many tasks are natural to specify with a sparse reward, and manually shaping a reward function can result in suboptimal performance. However, finding a non-zero reward is exponentially more difficult with increasing task horizon or action dimensionality. This puts many real-world tasks out of practical reach of RL methods. In this work, we use demonstrations to overcome the exploration problem and successfully learn to perform long-horizon, multi-step robotics tasks with continuous control such as stacking blocks with a robot arm. Our method, which builds on top of Deep Deterministic Policy Gradients and Hindsight Experience Replay, provides an order of magnitude of speedup over RL on simulated robotics tasks. It is simple to implement and makes only the additional assumption that we can collect a small set of demonstrations. Furthermore, our method is able to solve tasks not solvable by either RL or behavior cloning alone, and often ends up outperforming the demonstrator policy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Demonstration learning (DemoL) algorithms often used in robotics applications are DQfD [218], DDPGfD [219] and Q-filtered BC [220].”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“Building upon DDPG [124] and HER [182], Nair et al. (2018) [220] propose a method called Q-filtered Behaviour Cloning (QfBC) that also applies the BC loss to learn demonstration data to solve long-horizon, sparse-reward and goal-conditioned tasks.”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [220]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Probabilistic roadmaps for path planning in high-dimensional configuration spaces
Kavraki, L. E., Svestka, P., Latombe, J.-C., & Overmars, M. H. (1996) · IEEE Transactions on Robotics and Automation · doi:10.1109/70.508439 · Cited by 6,343 · cited 1×
“More recent work on task and motion planning considers both logical and physical aspects of the task [27], [28], [29], but requires domainspecific engineering.”
BackgroundMuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578
Self-citationEnd to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198
Apprenticeship learning via inverse reinforcement learning
Abbeel, P., & Ng, A. Y. (2004) · International Conference on Machine Learning · doi:10.1145/1015330.1015430 · Cited by 2,855
Self-citationUnderstanding natural language
Winograd, T. A. (1972) · Cognitive Psychology · doi:10.1016/0010-0285(72)90002-3 · Cited by 2,803
End-to-End Training of Deep Visuomotor PoliciesPreprint
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00702 · Cited by 2,641
Self-citationLevine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates
Gu, S., Holly, E., Lillicrap, T., & Levine, S. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989385 · Cited by 1,582
ALVINN, an autonomous land vehicle in a neural network
Pomerleau, D. (2018) · doi:10.1184/r1/6603146 · Cited by 1,369
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., & Zaremba, W. (2017) · arXiv · doi:10.48550/arxiv.1707.01495 · Cited by 1,355
Self-citationPILCO: A Model-Based and Data-Efficient Approach to Policy Search
Deisenroth, M. P., & Rasmussen, C. E. (2011) · Cambridge University Engineering Department Publications Database · Cited by 1,136
Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours
Pinto, L., & Gupta, A. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487517 · Cited by 1,099
Deep Q-learning From Demonstrations
Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., et al. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11757 · Cited by 889
Reinforcement learning of motor skills with policy gradients
Peters, J., & Schaal, S. (2008) · Neural Networks · doi:10.1016/j.neunet.2008.02.003 · Cited by 831
Hierarchical task and motion planning in the now
Kaelbling, L., & Lozano-Perez, T. (2011) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2011.5980391 · Cited by 712
Universal Value Function Approximators
Schaul, T., Horgan, D., Gregor, K., & Silver, D. (2015) · Cited by 701
Guided Cost Learning: Deep Inverse Optimal Control via Policy OptimizationPreprint
Finn, C., Levine, S., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.00448 · Cited by 683
Self-citationA Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots
Giusti, A., Guzzi, J., Ciresan, D. C., He, F.-L., Rodriguez, J. P., Fontana, F., Faessler, M., Forster, C., Schmidhuber, J., Caro, G. D., et al. (2016) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2015.2509024 · Cited by 665
Combined task and motion planning through an extensible planner-independent interface layer
Srivastava, S., Fang, E., Riano, L., Chitnis, R., Russell, S. J., & Abbeel, P. (2014) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2014.6906922 · Cited by 573
Self-citationRobot Learning from Demonstration
Atkeson, C. G., & Stefan Schaal, A. (2012) · SpringerReference · doi:10.1007/springerreference_302487 · Cited by 532
Relative Entropy Policy Search
Peters, J., Mülling, K., & Altün, Y. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7727 · Cited by 506
Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
Vecerik, M., Hester, T., Scholz, J., Wang, F., Pietquin, O., Piot, B., Heess, N., Rothörl, T., Lampe, T., & Riedmiller, M. (2017) · arXiv · doi:10.48550/arxiv.1707.08817 · Cited by 478
Policy search for motor primitives in robotics
Kober, J., & Peters, J. (2010) · Machine Learning · doi:10.1007/s10994-010-5223-6 · Cited by 477
One-Shot Imitation LearningPreprint
Duan, Y., Andrychowicz, M., Stadie, B. C., Ho, O. J., Schneider, J., Sutskever, I., Abbeel, P., & Zaremba, W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07326 · Cited by 473
Self-citationLearning from demonstration and adaptation of biped locomotion
Nakanishi, J., Morimoto, J., Endo, G., Cheng, G., Stefan Schaal, A., & Kawato, M. (2004) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2004.03.003 · Cited by 399
Reverse Curriculum Generation for Reinforcement LearningPreprint
Florensa, C., Held, D., Wulfmeier, M., Zhang, M., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05300 · Cited by 277
Self-citationLearning to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning
Deisenroth, M. P., Rasmussen, C. E., & Fox, D. (2011) · Robotics: Science and Systems · doi:10.15607/rss.2011.vii.008 · Cited by 208
Data-efficient Deep Reinforcement Learning for Dexterous ManipulationPreprint
Popov, I., Heess, N., Lillicrap, T., Hafner, R., Barth-Maron, G., Vecerík, M., Lampe, T., Tassa, Y., Erez, T., & Riedmiller, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03073 · Cited by 174
Learning locomotion over rough terrain using terrain templates
Kalakrishnan, M., Buchli, J., Pástor, P., & Schaal, S. (2009) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2009.5354701 · Cited by 90
Learning from Limited Demonstrations
Kim, B., Farahmand, A. M., Pineau, J., & Precup, D. (2013) · neural information processing systems · Cited by 76
Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy UpdatesPreprint
Gu, S., Holly, E., Lillicrap, T., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.00633 · Cited by 39
Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot HoursPreprint
Pinto, L., & Gupta, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.06825 · Cited by 28
Flagged first, then by citations
4 of the 44 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- hester, 2017, Learning from demonstrations for real world reinforcement learning
- ng, 2000, Algorithms for Inverse Reinforcement Learning, International Conference on Machine Learning (ICML)
- ziebart, 2008, Maximum Entropy Inverse Reinforcement Learning, AAAI Conference on Artificial Intelligence, 1433
Cited by
Sorted by citations
Grandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455
Continuous control for robot based on deep reinforcement learning
Zhang, S. (2019) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/90191 · Cited by 961
Peng, X. B., Abbeel, P., Levine, S., & van de Panne, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201311 · Cited by 946
Self-citationTransfer Learning in Deep Reinforcement Learning: A Survey
Zhu, Z., Lin, K., Jain, A. K., & Zhou, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292075 · Cited by 861
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., & Levine, S. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.049 · Cited by 847
Metadata
- Authors
- Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., & Abbeel, P.
- Affiliations
- OpenAI (United States) (US)
- ORCID
- Abbeel, Pieter
- Venue
- 2018 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2018-05
- Pages
- 6292-6299
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2756826236
- Cited by
- 712
- References
- 44
- Citations by year
- 2026: 24 · 2025: 72 · 2024: 86 · 2023: 111 · 2022: 93 · 2021: 117 · 2020: 105 · 2019: 67 · 2018: 35
- FWCI
- 33.08
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- exploration in reinforcement learning, sparse rewards, demonstrations, deep deterministic policy gradient, hindsight experience replay, long-horizon tasks, continuous control, robotic manipulation, block stacking, robot manipulators, behavior cloning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
