Overcoming Exploration in Reinforcement Learning with Demonstrations
Open accessAbstract
Exploration in environments with sparse rewards has been a persistent problem in reinforcement learning (RL). Many tasks are natural to specify with a sparse reward, and manually shaping a reward function can result in suboptimal performance. However, finding a non-zero reward is exponentially more difficult with increasing task horizon or action dimensionality. This puts many real-world tasks out of practical reach of RL methods. In this work, we use demonstrations to overcome the exploration problem and successfully learn to perform long-horizon, multi-step robotics tasks with continuous control such as stacking blocks with a robot arm. Our method, which builds on top of Deep Deterministic Policy Gradients and Hindsight Experience Replay, provides an order of magnitude of speedup over RL on simulated robotics tasks. It is simple to implement and makes only the additional assumption that we can collect a small set of demonstrations. Furthermore, our method is able to solve tasks not solvable by either RL or behavior cloning alone, and often ends up outperforming the demonstrator policy.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Demonstration learning algorithms often used in robotics applications are DQfD, DDPGfD and Q-filtered BC.
“Demonstration learning (DemoL) algorithms often used in robotics applications are DQfD [218], DDPGfD [219] and Q-filtered BC [220].”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [220]
QfBC applies the BC loss to learn demonstration data to solve long-horizon, sparse-reward and goal-conditioned tasks.
“Building upon DDPG [124] and HER [182], Nair et al. (2018) [220] propose a method called Q-filtered Behaviour Cloning (QfBC) that also applies the BC loss to learn demonstration data to solve long-horizon, sparse-reward and goal-conditioned tasks.”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [220]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · see citing works
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · see citing works
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · see citing works
Probabilistic roadmaps for path planning in high-dimensional configuration spaces
Kavraki, L. E., Svestka, P., Latombe, J.-C., & Overmars, M. H. (1996) · IEEE Transactions on Robotics and Automation · doi:10.1109/70.508439 · Cited by 6,343 · cited 1× · see citing works
“More recent work on task and motion planning considers both logical and physical aspects of the task [27], [28], [29], but requires domainspecific engineering.”
BackgroundMuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · see citing works
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578 · see citing works
Self-citationEnd-to-End Training of Deep Visuomotor PoliciesPreprint
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2015) · Journal of machine learning research · doi:10.48550/arxiv.1504.00702 · Cited by 4,011 · see citing works
Self-citationEnd to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198 · see citing works
Apprenticeship learning via inverse reinforcement learning
Abbeel, P., & Ng, A. Y. (2004) · International Conference on Machine Learning · doi:10.1145/1015330.1015430 · Cited by 2,855 · see citing works
Self-citationUnderstanding natural language
Winograd, T. A. (1972) · Cognitive Psychology · doi:10.1016/0010-0285(72)90002-3 · Cited by 2,803 · see citing works
End-to-end training of deep visuomotor policies
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2016) · Journal of Machine Learning Research · Cited by 2,192 · see citing works
Self-citationLevine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971 · see citing works
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941 · see citing works
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates
Gu, S., Holly, E., Lillicrap, T., & Levine, S. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989385 · Cited by 1,485 · see citing works
ALVINN, an autonomous land vehicle in a neural network
Pomerleau, D. A. (2018) · doi:10.1184/r1/6603146 · Cited by 1,403 · see citing works
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., & Zaremba, W. (2017) · arXiv · doi:10.48550/arxiv.1707.01495 · Cited by 1,355 · see citing works
Self-citationPILCO: A Model-Based and Data-Efficient Approach to Policy Search
Deisenroth, M. P., & Rasmussen, C. E. (2011) · Cambridge University Engineering Department Publications Database · Cited by 1,136 · see citing works
Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours
Pinto, L., & Gupta, A. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487517 · Cited by 1,060 · see citing works
Deep Q-learning From Demonstrations
Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., et al. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11757 · Cited by 889 · see citing works
Reinforcement learning of motor skills with policy gradients
Peters, J., & Schaal, S. (2008) · Neural Networks · doi:10.1016/j.neunet.2008.02.003 · Cited by 831 · see citing works
Hierarchical task and motion planning in the now
Kaelbling, L., & Lozano-Perez, T. (2011) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2011.5980391 · Cited by 712 · see citing works
Universal Value Function Approximators
Schaul, T., Horgan, D., Gregor, K., & Silver, D. (2015) · Cited by 677 · see citing works
A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots
Giusti, A., Guzzi, J., Ciresan, D. C., He, F.-L., Rodriguez, J. P., Fontana, F., Faessler, M., Forster, C., Schmidhuber, J., Caro, G. D., et al. (2016) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2015.2509024 · Cited by 665 · see citing works
Guided Cost Learning: Deep Inverse Optimal Control via Policy OptimizationPreprint
Finn, C., Levine, S., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.00448 · Cited by 589 · see citing works
Self-citationRobot Learning from Demonstration
Atkeson, C. G., & Stefan Schaal, A. (2012) · SpringerReference · doi:10.1007/springerreference_302487 · Cited by 532 · see citing works
Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
Vecerik, M., Hester, T., Scholz, J., Wang, F., Pietquin, O., Piot, B., Heess, N., Rothörl, T., Lampe, T., & Riedmiller, M. (2017) · arXiv · doi:10.48550/arxiv.1707.08817 · Cited by 478 · see citing works
Policy search for motor primitives in robotics
Kober, J., & Peters, J. (2010) · Machine Learning · doi:10.1007/s10994-010-5223-6 · Cited by 477 · see citing works
Relative Entropy Policy Search
Peters, J., Mülling, K., & Altün, Y. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7727 · Cited by 467 · see citing works
Combined task and motion planning through an extensible planner-independent interface layer
Srivastava, S., Fang, E., Riano, L., Chitnis, R., Russell, S., & Abbeel, P. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6906922 · Cited by 466 · see citing works
Self-citationOne-Shot Imitation LearningPreprint
Duan, Y., Andrychowicz, M., Stadie, B. C., Ho, O. J., Schneider, J., Sutskever, I., Abbeel, P., & Zaremba, W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07326 · Cited by 449 · see citing works
Self-citationLearning from demonstration and adaptation of biped locomotion
Nakanishi, J., Morimoto, J., Endo, G., Cheng, G., Stefan Schaal, A., & Kawato, M. (2004) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2004.03.003 · Cited by 399 · see citing works
Reverse Curriculum Generation for Reinforcement LearningPreprint
Florensa, C., Held, D., Wulfmeier, M., Zhang, M., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05300 · Cited by 277 · see citing works
Self-citationLearning to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning
Deisenroth, M. P., Rasmussen, C. E., & Fox, D. (2011) · Robotics: Science and Systems · doi:10.15607/rss.2011.vii.008 · Cited by 208 · see citing works
Data-efficient Deep Reinforcement Learning for Dexterous ManipulationPreprint
Popov, I., Heess, N., Lillicrap, T., Hafner, R., Barth-Maron, G., Vecerík, M., Lampe, T., Tassa, Y., Erez, T., & Riedmiller, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03073 · Cited by 174 · see citing works
Learning locomotion over rough terrain using terrain templates
Kalakrishnan, M., Buchli, J., Pástor, P., & Stefan Schaal, A. (2009) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2009.5354701 · Cited by 87 · see citing works
Learning from Limited Demonstrations
Kim, B., Farahmand, A. M., Pineau, J., & Precup, D. (2013) · neural information processing systems · Cited by 76 · see citing works
Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy UpdatesPreprint
Gu, S., Holly, E., Lillicrap, T., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.00633 · Cited by 39 · see citing works
Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot HoursPreprint
Pinto, L., & Gupta, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.06825 · Cited by 28 · see citing works
Flagged first, then by citations
3 of the 44 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- A. Ng and S. Russell, "Algorithms for Inverse Reinforcement Learn- ing," International Conference on Machine Learning (ICML), 2000.
- B. D. Ziebart et al., "Maximum Entropy Inverse Reinforcement Learning." in AAAI Conference on Artificial Intelligence, 2008, pp. 1433-1438.
- T. Hester et al., "Learning from Demonstrations for Real World Reinforcement Learning," arXiv preprint arxiv:1704.03732, 2017. · arXiv
Cited by
Sorted by citations
Grandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · see citing works
Continuous control for robot based on deep reinforcement learning
Zhang, S. (2019) · doi:10.32657/10356/90191 · Cited by 953 · see citing works
Peng, X. B., Abbeel, P., Levine, S., & van de Panne, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201311 · Cited by 946 · see citing works
Self-citationTransfer Learning in Deep Reinforcement Learning: A Survey
Zhu, Z., Lin, K., Jain, A. K., & Zhou, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292075 · Cited by 861 · see citing works
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., & Levine, S. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.049 · Cited by 847 · see citing works
Recent Advances in Robot Learning from Demonstration
Ravichandar, H., Polydoros, A. S., Chernova, S., & Billard, A. (2020) · Annu. Rev. Control. Robotics Auton. Syst. · doi:10.1146/annurev-control-100819-063206 · Cited by 947 · see citing works
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570 · see citing works
Exploration in deep reinforcement learning: A survey
Ladosz, P., Weng, L., Kim, M., & Oh, H. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.003 · Cited by 525 · see citing works
Off-Policy Deep Reinforcement Learning without ExplorationPreprint
Fujimoto, S., Meger, D., & Precup, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.02900 · Cited by 533 · see citing works
Residual Reinforcement Learning for Robot Control
Johannink, T., Bahl, S., Nair, A., Luo, J., Kumar, A., Loskyll, M., Ojea, J. A., Solowjow, E., & Levine, S. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8794127 · Cited by 480 · see citing works
Self-citationPRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning
Sartoretti, G., Kerr, J. H., Shi, Y., Wagner, G., Kumar, T. K. S., Koenig, S., & Choset, H. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2903261 · Cited by 426 · see citing works
Learning by Playing - Solving Sparse Reward Tasks from ScratchPreprint
Riedmiller, M., Hafner, R., Lampe, T., Neunert, M., Degrave, J., Van de Wiele, T., Mnih, V., Heess, N., & Springenberg, J. T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10567 · Cited by 314 · see citing works
Deep reinforcement learning in computer vision: a comprehensive survey
Le, N., Rathour, V. S., Yamazaki, K., Luu, K., & Savvides, M. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-10061-9 · Cited by 265 · see citing works
Reinforcement and Imitation Learning for Diverse Visuomotor Skills
Zhu, Y., Wang, Z., Merel, J., Rusu, A. A., Erez, T., Cabi, S., Tunyasuvunakool, S., Kramár, J., Hadsell, R. T., de Freitas, N., et al. (2018) · Robotics: Science and Systems · doi:10.15607/rss.2018.xiv.009 · Cited by 267 · see citing works
A Minimalist Approach to Offline Reinforcement LearningPreprint
Fujimoto, S., & Gu, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06860 · Cited by 267 · see citing works
A Survey on Deep Reinforcement Learning Algorithms for Robotic Manipulation
Han, D., Mulyana, B., Stankovic, V., & Cheng, S. (2023) · Sensors · doi:10.3390/s23073762 · Cited by 253 · see citing works
Deep reinforcement learning in medical imaging: A literature review
Zhou, S., Le, N., Luu, K., Nguyen, H. V., & Ayache, N. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102193 · Cited by 222 · see citing works
Wu, J., Huang, Z., Hu, Z., & Lv, C. (2023) · Engineering · doi:10.1016/j.eng.2022.05.017 · Cited by 205 · see citing works
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J. (2021) · RePEc: Research Papers in Economics · Cited by 201 · see citing works
Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards
Schoettler, G., Nair, A., Luo, J., Bahl, S., Ojea, J. A., Solowjow, E., & Levine, S. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341714 · Cited by 191 · see citing works
Self-citationVariable Compliance Control for Robotic Peg-in-Hole Assembly: A Deep-Reinforcement-Learning Approach
Beltran-Hernandez, C. C., Petit, D., Ramírez-Alpizar, I. G., & Harada, K. (2020) · Applied Sciences · doi:10.3390/app10196923 · Cited by 189 · see citing works
Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost
Zhu, H., Gupta, A., Rajeswaran, A., Levine, S., & Kumar, V. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794102 · Cited by 184 · see citing works
Decentralized Multi-Agent Pursuit Using Deep Reinforcement Learning
de Souza, C., Newbury, R., Cosgun, A., Castillo, P., Vidolov, B., & Kuli, D. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3068952 · Cited by 184 · see citing works
Xu, B., Rathod, D., Zhang, D., Yebi, A., Zhang, X., Li, X., & Filipi, Z. S. (2019) · Applied Energy · doi:10.1016/j.apenergy.2019.114200 · Cited by 182 · see citing works
Retzlaff, C. O., Das, S., Wayllace, C., Mousavi, P., Afshari, M. P., Yang, T., Saranti, A., Angerschmid, A., Taylor, M. E., & Holzinger, A. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15348 · Cited by 194 · see citing works
AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
Nair, A., Gupta, A., Dalal, M., & Levine, S. (2020) · arXiv · doi:10.48550/arxiv.2006.09359 · Cited by 175 · see citing works
Self-citationReinforcement Learning for Mobile Robotics Exploration: A Survey
Garaffa, L. C., Basso, M., Konzen, A. A., & de Freitas, E. P. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3124466 · Cited by 178 · see citing works
Deep Reinforcement Learning for Traffic Signal Control: A Review
Rasheed, F., Yau, K. A., Noor, R. M., Wu, C., & Low, Y.-C. (2020) · IEEE Access · doi:10.1109/access.2020.3034141 · Cited by 181 · see citing works
Deep learning in robotics: survey on model structures and training strategies
Karoly, A. I., Galambos, P., Kuti, J., & Rudas, I. J. (2021) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2020.3018325 · Cited by 146 · see citing works
Dorokhova, M., Martinson, Y., Ballif, C., & Wyrsch, N. (2021) · Applied Energy · doi:10.1016/j.apenergy.2021.117504 · Cited by 143 · see citing works
Adaptive Quantitative Trading: An Imitative Deep Reinforcement Learning Approach
Liu, Y., Liu, Q., Zhao, H., Pan, Z., & Liu, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i02.5587 · Cited by 141 · see citing works
ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation
Mandlekar, A., Zhu, Y., Garg, A., Booher, J., Spero, M., Tung, A., Gao, J., Emmons, J., Gupta, A., Orbay, E., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.02790 · Cited by 134 · see citing works
Deep Reinforcement Learning for Soft, Flexible Robots: Brief Review with Impending Challenges
Bhagat, S., Banerjee, H., Tse, Z. T. H., & Ren, H. (2019) · Robotics · doi:10.3390/robotics8010004 · Cited by 134 · see citing works
Wang, Z., Novikov, A., Zolna, K., Springenberg, J. T., Reed, S., Shahriari, B., Siegel, N., Merel, J., Gulcehre, C., Heess, N., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.15134 · Cited by 128 · see citing works
Prioritized Experience-Based Reinforcement Learning With Human Guidance for Autonomous Driving
Wu, J., Huang, Z., Huang, W., & Lv, C. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3177685 · Cited by 123 · see citing works
Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration
Johns, E. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9560942 · Cited by 119 · see citing works
Goal-Conditioned Imitation LearningPreprint
Ding, Y., Florensa, C., Phielipp, M., & Abbeel, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05838 · Cited by 119 · see citing works
Self-citationSearch on the Replay Buffer: Bridging Planning and Reinforcement Learning
Eysenbach, B., Salakhutdinov, R., & Levine, S. (2019) · arXiv · doi:10.48550/arxiv.1906.05253 · Cited by 123 · see citing works
A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learning
Vecerík, M., Sushkov, O. O., Barker, D., Rothörl, T., Hester, T., & Scholz, J. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794074 · Cited by 113 · see citing works
Reward learning from human preferences and demonstrations in Atari
Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., & Amodei, D. (2018) · arXiv · doi:10.48550/arxiv.1811.06521 · Cited by 117 · see citing works
Goal-Conditioned Reinforcement Learning: Problems and Solutions
Liu, M., Zhu, M., & Zhang, W. (2022) · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/770 · Cited by 115 · see citing works
“Researchers used to evaluate their image algorithms on simulation environments, as in Warde-Farley et al. ; Lee et al. ; Chane-Sane et al. [2019; 2020; 2021], and also in real-world robotics applications [Nair et al. , 2018b; Nair et al. , 2020b].”
BackgroundMethodDynamic Cloth Manipulation with Deep Reinforcement Learning
Jangir, R., Alenya, G., & Torras, C. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9196659 · Cited by 114 · see citing works
Human-Guided Reinforcement Learning With Sim-to-Real Transfer for Autonomous Navigation
Wu, J., Zhou, Y., Yang, H., Huang, Z., & Lv, C. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3314762 · Cited by 110 · see citing works
Model-free reinforcement learning from expert demonstrations: a survey
Ramírez, J., Yu, W., & Perrusquía, A. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-10085-1 · Cited by 110 · see citing works
Xu, J., Li, B., Lu, B., Liu, Y., Dou, Q., & Heng, P. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9635867 · Cited by 107 · see citing works
GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning AlgorithmsPreprint
Colas, C., Sigaud, O., & Oudeyer, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.05054 · Cited by 118 · see citing works
Mohammed, M. Q., Kwek, L. C., & Chua, S. C. (2020) · IEEE Access · doi:10.1109/access.2020.3027923 · Cited by 108 · see citing works
Privacy Preserving Load Control of Residential Microgrid via Deep Reinforcement Learning
Qin, Z., Liu, D., Hua, H., & Cao, J. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2021.3088290 · Cited by 105 · see citing works
CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement LearningPreprint
Colas, C., Fournier, P., Sigaud, O., Chétouani, M., & Oudeyer, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.06284 · Cited by 104 · see citing works
Policy Optimization with Demonstrations
Kang, B., Jie, Z., & Feng, J. (2018) · International Conference on Machine Learning · Cited by 101 · see citing works
Nguyen, K., Dey, D., Brockett, C., & Dolan, B. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01281 · Cited by 101 · see citing works
Hierarchical Reinforcement Learning With Universal Policies for Multistep Robotic Manipulation
Yang, X., Ji, Z., Wu, J., Lai, Y., Wei, C., Liu, G., & Setchi, R. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3059912 · Cited by 94 · see citing works
Hierarchical Imitation and Reinforcement LearningPreprint
Le, H. M., Jiang, N., Agarwal, A., Dudı́k, M., Yue, Y., & Daumé, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.00590 · Cited by 94 · see citing works
Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks
Nasiriany, S., Liu, H., & Zhu, Y. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9812140 · Cited by 91 · see citing works
Huang, Z., Sheng, Z., Ma, C., & Chen, S. (2024) · Communications in Transportation Research · doi:10.1016/j.commtr.2024.100127 · Cited by 88 · see citing works
Meta-Reinforcement Learning for Robotic Industrial Insertion Tasks
Schoettler, G., Nair, A., Ojea, J. A., Levine, S., & Solowjow, E. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9340848 · Cited by 88 · see citing works
Self-citationLuo, J., Sushkov, O. O., Pevceviciute, R., Lian, W., Su, C., Vecerík, M., Ye, N., Stefan Schaal, A., & Scholz, J. (2021) · Robotics: Science and Systems · doi:10.15607/rss.2021.xvii.088 · Cited by 88 · see citing works
Learning Montezuma's Revenge from a Single DemonstrationPreprint
Salimans, T., & Chen, R. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.03381 · Cited by 102 · see citing works
Liu, B., Wang, L., Liu, M., & Xu, C. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2976321 · Cited by 83 · see citing works
Thananjeyan, B., Balakrishna, A., Rosolia, U., Li, F., McAllister, R., Gonzalez, J. E., Levine, S., Borrelli, F., & Goldberg, K. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2976272 · Cited by 83 · see citing works
A review of recent trend in motion planning of industrial robots
Tamizi, M. G., Yaghoubi, M., & Najjaran, H. (2023) · International Journal of Intelligent Robotics and Applications · doi:10.1007/s41315-023-00274-2 · Cited by 83 · see citing works
Observe and Look Further: Achieving Consistent Performance on AtariPreprint
Pohlen, T., Piot, B., Hester, T., Azar, M. G., Horgan, D., Budden, D. M., Barth-Maron, G., van Hasselt, H. P., Quan, J., Vecerík, M., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11593 · Cited by 81 · see citing works
Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation
Antotsiou, D., Garcia-Hernando, G., & Kim, T. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11024-6_19 · Cited by 80 · see citing works
Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
Li, R., Jabri, A., Darrell, T., & Agrawal, P. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197468 · Cited by 79 · see citing works
Mandlekar, A., Ramos, F. T., Boots, B., Savarese, S., Fei-Fei, L., Garg, A., & Fox, D. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196935 · Cited by 79 · see citing works
Deep Reinforcement Learning With Optimized Reward Functions for Robotic Trajectory Planning
Xie, J., Shao, Z., Li, Y., Guan, Y., & Tan, J. (2019) · IEEE Access · doi:10.1109/access.2019.2932257 · Cited by 78 · see citing works
Keller, B., Draelos, M., Zhou, K. C., Qian, R., Kuo, A. N., Konidaris, G., Hauser, K. K., & Izatt, J. A. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2020.2980158 · Cited by 77 · see citing works
Colas, C., Karch, T., Sigaud, O., & Oudeyer, P. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13554 · Cited by 101 · see citing works
VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation
Hoque, R., Seita, D., Balakrishna, A., Ganapathi, A., Tanwani, A., Jamali, N., Yamane, K., Iba, S., & Goldberg, K. (2020) · Robotics: Science and Systems XVI · doi:10.15607/rss.2020.xvi.034 · Cited by 74 · see citing works
Lee, H.-R., & Lee, T. (2020) · European Journal of Operational Research · doi:10.1016/j.ejor.2020.09.018 · Cited by 72 · see citing works
AGIL: Learning Attention from Human for Visuomotor Tasks
Zhang, R., Liu, Z., Zhang, L., Whritner, J. A., Muller, K., Hayhoe, M., & Ballard, D. H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_41 · Cited by 71 · see citing works
Xiang, X., & Foo, S. Y. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3030029 · Cited by 72 · see citing works
Q-Attention: Enabling Efficient Learning for Vision-Based Robotic Manipulation
James, S., & Davison, A. J. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3140817 · Cited by 68 · see citing works
Open-Sourced Reinforcement Learning Environments for Surgical Robotics
Richter, F., Orosco, R. K., & Yip, M. C. (2019) · arXiv · doi:10.48550/arxiv.1903.02090
Improvisation through Physical Understanding: Using Novel Objects As Tools with Visual Foresight
Xie, A., Ebert, F., Levine, S., & Finn, C. (2019) · Robotics: Science and Systems · doi:10.15607/rss.2019.xv.001 · Cited by 66 · see citing works
Deep Learning in Diverse Intelligent Sensor Based Systems
Zhu, Y., Wang, M., Yin, X., Zhang, J., Meijering, E. H. W., & Hu, J. (2022) · Sensors · doi:10.3390/s23010062 · Cited by 67 · see citing works
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to ImitationPreprint
Tai, L., Zhang, J., Liu, M., Boedecker, J., & Burgard, W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.07139 · Cited by 95 · see citing works
Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving
Cao, Z., Bıyık, E., Wang, W. Z., Raventos, A., Gaidon, A., Rosman, G., & Sadigh, D. (2020) · Robotics: Science and Systems · doi:10.15607/rss.2020.xvi.039 · Cited by 65 · see citing works
Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planning
Chiu, Z.-Y., Richter, F., Funk, E. K., Orosco, R. K., & Yip, M. C. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561673 · Cited by 62 · see citing works
Self-Supervised Sim-to-Real Adaptation for Visual Robotic Manipulation
Jeong, R., Aytar, Y., Khosid, D., Zhou, Y., Kay, J., Lampe, T., Bousmalis, K., & Nori, F. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197326 · Cited by 60 · see citing works
Model-based Deep Reinforcement Learning for Dynamic Portfolio OptimizationPreprint
Yu, P., Sern, L. J., Kulyatin, I., Shi, Z., & Dasgupta, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08740 · Cited by 82 · see citing works
Robot learning from demonstration for path planning: A review
Xie, Z., Zhang, Q., Jiang, Z., & Liu, H. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1648-4 · Cited by 58 · see citing works
Xie, Y., & Zhao, X. (2021) · Physics of Fluids · doi:10.1063/5.0037334 · Cited by 57 · see citing works
From Model-Centric to Data-Centric AI: A Paradigm Shift or Rather a Complementary Approach?
Hamid, O. H. (2022) · International Conference on Information Technology Trends (ITT) · doi:10.1109/itt56123.2022.9863935 · Cited by 57 · see citing works
Lee, M. A., Florensa, C., Tremblay, J., Ratliff, N., Garg, A., Ramos, F. T., & Fox, D. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197125 · Cited by 57 · see citing works
Exact-K Recommendation via Maximal Clique Optimization
Gong, Y., Zhu, Y., Duan, L., Liu, Q., Guan, Z., Sun, F., Ou, W., & Zhu, K. Q. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330832 · Cited by 56 · see citing works
Reinforcement Learning for Statistical Process Control in Manufacturing
Viharos, Z. J., & Jakab, R. (2021) · Measurement · doi:10.1016/j.measurement.2021.109616 · Cited by 55 · see citing works
Surgical embodied intelligence for generalized task autonomy in laparoscopic robot-assisted surgery
Long, Y., Lin, A., Kwok, D. H. C., Zhang, L., Yang, Z., Shi, K., Song, L., Fu, J., Lin, H., Wang, W., et al. (2025) · Science Robotics · doi:10.1126/scirobotics.adt3093 · Cited by 55 · see citing works
Task-Oriented Deep Reinforcement Learning for Robotic Skill Acquisition and Control
Xiang, G., & Su, J. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2949596 · Cited by 54 · see citing works
Accelerating Reinforcement Learning for Reaching Using Continuous Curriculum Learning
Luo, S., Kasaei, H., & Schomaker, L. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207427 · Cited by 56 · see citing works
Divide-and-Conquer Reinforcement LearningPreprint
Ghosh, D., Singh, A., Rajeswaran, A., Kumar, V., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.09874 · Cited by 53 · see citing works
Parrot: Data-Driven Behavioral Priors for Reinforcement LearningPreprint
Singh, A., Liu, H., Zhou, G., Yu, A., Rhinehart, N., & Levine, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.10024 · Cited by 57 · see citing works
Li, X., Liang, X., Wang, X., Wang, R., Shu, L., & Xu, W. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110543 · Cited by 51 · see citing works
Bayesian Gaussian Mixture Model for Robotic Policy Imitation
Pignat, E., & Calinon, S. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2932610 · Cited by 50 · see citing works
Overcoming model bias for robust offline deep reinforcement learning
Swazinna, P., Udluft, S., & Runkler, T. A. (2021) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2021.104366 · Cited by 52 · see citing works
Making Efficient Use of Demonstrations to Solve Hard Exploration ProblemsPreprint
Paine, T. L., Gülçehre, Ç., Shahriari, B., Denil, M., Hoffman, M., Soyer, H., Tanburn, R., Kapturowski, S., Rabinowitz, N. C., Williams, D., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01387 · Cited by 49 · see citing works
Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors
Wang, L., Liu, J., Shao, H., Wang, W., Chen, R., Liu, Y., & Waslander, S. L. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.102 · Cited by 52 · see citing works
Rewards Prediction-Based Credit Assignment for Reinforcement Learning With Sparse Binary Rewards
Seo, M., Vecchietti, L. F., Lee, S., & Har, D. (2019) · IEEE Access · doi:10.1109/access.2019.2936863 · Cited by 47 · see citing works
DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable FeedbackPreprint
Arakawa, R., Kobayashi, S., Unno, Y., Tsuboi, Y., & Maeda, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.11748 · Cited by 47 · see citing works
Niu, H., Li, S., & Li, J. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557363 · Cited by 46 · see citing works
- Authors
- Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., & Abbeel, P.
- Affiliations
- OpenAI (United States) (US)
- ORCID
- Abbeel, Pieter
- Venue
- 2018 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2018-05
- Pages
- 6292-6299
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2756826236
- Cited by
- 712
- References
- 44
- Citations by year
- 2026: 24 · 2025: 72 · 2024: 86 · 2023: 111 · 2022: 93 · 2021: 117 · 2020: 105 · 2019: 67 · 2018: 35 · 2017: 1 · 2016: 1
- FWCI
- 33.08
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- exploration in reinforcement learning, sparse rewards, demonstrations, deep deterministic policy gradient, hindsight experience replay, long-horizon tasks, continuous control, robotic manipulation, block stacking, robot manipulators, behavior cloning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
