Deep Q-learning From Demonstrations
Open accessAbstract
Deep reinforcement learning (RL) has achieved several high profile successes in difficult decision-making problems. However, these algorithms typically require a huge amount of data before they reach reasonable performance. In fact, their performance during learning can be extremely poor. This may be acceptable for a simulator, but it severely limits the applicability of deep RL to many real-world tasks, where the agent must learn in the real environment. In this paper we study a setting where the agent may access data from previous control of the system. We present an algorithm, Deep Q-learning from Demonstrations (DQfD), that leverages small sets of demonstration data to massively accelerate the learning process even from relatively small amounts of demonstration data and is able to automatically assess the necessary ratio of demonstration data while learning thanks to a prioritized replay mechanism. DQfD works by combining temporal difference updates with supervised classification of the demonstrator’s actions. We show that DQfD has better initial performance than Prioritized Dueling Double Deep Q-Networks (PDD DQN) as it starts with better scores on the first million steps on 41 of 42 games and on average it takes PDD DQN 83 million steps to catch up to DQfD’s performance. DQfD learns to out-perform the best demonstration given in 14 of 42 games. In addition, DQfD leverages human demonstrations to achieve state-of-the-art results for 11 games. Finally, we show that DQfD performs better than three related algorithms for incorporating demonstration data into DQN.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
DQfD has outperformed its pure RL and IL counterparts on most of the Atari games.
“DQfN has outperformed its pure RL and IL counterparts on most of the Atari games.”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [218]
The Results (p. 5) report that DQfD beat pure RL (PDD DQN) on 41 of 42 games during the first million steps and started ahead of pure imitation learning on 31 games. Those comparisons support an early-performance advantage on most games, but the citing sentence omits that time qualification and implies an overall advantage. (LLM, conf=0.87)
Experimental — accuracy not yet measuredDemonstration learning algorithms often used in robotics applications are DQfD, DDPGfD and Q-filtered BC.
“Demonstration learning (DemoL) algorithms often used in robotics applications are DQfD [218], DDPGfD [219] and Q-filtered BC [220].”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [218]
The Abstract (p. 1) identifies DQfD as an algorithm that learns from demonstrations, supporting its inclusion in the list. The Discussion (p. 6) describes potential real-world settings, including autonomous vehicles, but the paper evaluates DQfD on Atari games (Experimental Setup, p. 4); it does not establish that DQfD is often used in robotics applications. DDPGfD and Q-filtered BC have separate citations and are not assessed against this source. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · see citing works
Self-citationAsynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · see citing works
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855 · see citing works
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318 · see citing works
Dueling Network Architectures for Deep Reinforcement Learning
Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & de Freitas, N. (2015) · arXiv · doi:10.48550/arxiv.1511.06581 · Cited by 3,300 · see citing works
Self-citationSchaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015) · arXiv · doi:10.48550/arxiv.1511.05952 · Cited by 3,120 · see citing works
Self-citationApprenticeship learning via inverse reinforcement learning
Abbeel, P., & Ng, A. Y. (2004) · doi:10.1145/1015330.1015430 · Cited by 2,873 · see citing works
End-to-end training of deep visuomotor policies
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2016) · Journal of Machine Learning Research · Cited by 2,192 · see citing works
Generative adversarial imitation learning
Ho, J., & Ermon, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03476 · Cited by 1,942 · see citing works
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941 · see citing works
(2000) · doi:10.2460/ajvr.67.2.323 · Cited by 1,621 · see citing works
Reinforcement Learning with Unsupervised Auxiliary TasksPreprint
Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05397 · Cited by 826 · see citing works
Self-citationGuided Cost Learning: Deep Inverse Optimal Control via Policy OptimizationPreprint
Finn, C., Levine, S., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.00448 · Cited by 589 · see citing works
An Application of Reinforcement Learning to Aerobatic Helicopter Flight
Abbeel, P., Coates, A., Quigley, M., & Ng, A. Y. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/7503.003.0006 · Cited by 551 · see citing works
One-Shot Imitation LearningPreprint
Duan, Y., Andrychowicz, M., Stadie, B. C., Ho, O. J., Schneider, J., Sutskever, I., Abbeel, P., & Zaremba, W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07326 · Cited by 449 · see citing works
Sammut, C., & Webb, G. I. (2010) · Encyclopedia of Machine Learning · doi:10.1007/978-0-387-30164-8_447 · Cited by 400 · see citing works
Count-Based Exploration with Neural Density Models
Ostrovski, G., Bellemare, M. G., Oord, A. V. D., & Munos, R. (2017) · arXiv · doi:10.48550/arxiv.1703.01310 · Cited by 377 · see citing works
Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw ImagesPreprint
Watter, M., Springenberg, J. T., Boedecker, J., & Riedmiller, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.07365 · Cited by 296 · see citing works
A Game-Theoretic Approach to Apprenticeship Learning
Umar, S. M., & Schapire, R. E. (2007) · Cited by 218 · see citing works
Apprenticeship learning using linear programming
Syed, U., Bowling, M., & Schapire, R. E. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390286 · Cited by 188 · see citing works
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2012) · arXiv · doi:10.48550/arxiv.1207.4708 · Cited by 187 · see citing works
Reinforcement learning from demonstration through shaping
Brys, T., Harutyunyan, A., Suay, H. B., Chernova, S., Taylor, M. E., & Nowé, A. (2015) · Cited by 174 · see citing works
Apprenticeship Learning About Multiple Intentions
Babes, M., Marivate, V. N., Subramanian, K. A., & Littman, M. L. (2011) · International Conference on Machine Learning · Cited by 136 · see citing works
Integrating reinforcement learning with human demonstrations of varying ability
Taylor, M. E., Suay, H. B., & Chernova, S. (2011) · doi:10.5555/2031678.2031705 · Cited by 131 · see citing works
Learning from Demonstrations for Real World Reinforcement LearningPreprint
Hester, T., Vecerík, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Sendonaris, A., Dulac-Arnold, G., Osband, I., Agapiou, J., et al. (2017) · arXiv (Cornell University) · Cited by 92 · see citing works
Self-citationLearning values across many orders of magnitudePreprint
van Hasselt, H. P., Guez, A., Hessel, M., Mnih, V., & Silver, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.07714 · Cited by 84 · see citing works
Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential PredictionPreprint
Sun, W. Y., Venkatraman, A., Gordon, G. J., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01030 · Cited by 80 · see citing works
Learning from Limited Demonstrations
Kim, B., Farahmand, A. M., Pineau, J., & Precup, D. (2013) · neural information processing systems · Cited by 76 · see citing works
Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint ReplayPreprint
Hosu, I.-A., & Rebedea, T. E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.05077 · Cited by 71 · see citing works
Exploration from Demonstration for Interactive Reinforcement Learning
Subramanian, K. A., Isbell, C. L., & Thomaz, A. L. (2016) · Adaptive Agents and Multi-Agents Systems · doi:10.5555/2936924.2936990 · Cited by 68 · see citing works
Policy shaping with human teachers
Cederborg, T., Grover, I., Isbell, C. L., & Thomaz, A. L. (2015) · Cited by 66 · see citing works
Boosted Bellman Residual Minimization Handling Expert Demonstrations
Piot, B., Geist, M., & Pietquin, O. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44851-9_35 · Cited by 63 · see citing works
Self-citationLearning from Demonstration for Shaping through Inverse Reinforcement Learning
Suay, H. B., Brys, T., Taylor, M. E., & Chernova, S. (2016) · VUBIR (Vrije Universiteit Brussel) · doi:10.5555/2936924.2936988 · Cited by 62 · see citing works
Efficient Exploration for Dialog Policy Learning with Deep BBQ Networks \& Replay Buffer Spiking.Preprint
Lipton, Z. C., Gao, J., Li, L. C., Li, X., Ahmed, F., & Deng, L. (2016) · arXiv (Cornell University) · Cited by 38 · see citing works
(2012) · Harvard University Press eBooks · doi:10.2307/j.ctt2jbq07.6 · Cited by 32 · see citing works
The Reactor: A Sample-Efficient Actor-Critic Architecture
Gruslys, A., Azar, M. G., Bellemare, M. G., & Munos, R. (2017) · Cited by 31 · see citing works
Self-citationLearning from Demonstrations: Is It Worth Estimating a Reward Function?
Piot, B., Geist, M., & Pietquin, O. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40988-2_2 · Cited by 26 · see citing works
Self-citationDirect Policy Iteration with Demonstrations
Chemali, J. J., & Lazaric, A. (2015) · LillOA (Université de Lille (University Of Lille)) · Cited by 23 · see citing works
Boosted and reward-regularized classification for apprenticeship learning
Piot, B., Geist, M., & Pietquin, O. (2014) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/jsmq1483 · Cited by 20 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 4
- Gruslys, A.; Gheshlaghi Azar, M.; Bellemare, M. G.; and Munos, R. 2017. The Reactor: A Sample-Efficient ActorCritic Architecture. ArXiv e-prints .
- Lakshminarayanan, A. S.; Ozair, S.; and Bengio, Y. 2016. Reinforcement learning with few expert demonstrations. In NIPS Workshop on Deep Learning for Action and Interaction .
- Schaal, S. 1996. Learning from demonstration. In Advances in Neural Information Processing Systems (NIPS) .
- Shani, G.; Heckerman, D.; and Brafman, R. I. 2005. An mdpbased recommender system. Journal of Machine Learning Research 6:1265-1295.
Cited by
Sorted by citations
Deep reinforcement learning for autonomous driving: A survey
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A. A. A., Yogamani, S., & Perez, P. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3054625 · Cited by 2,342 · see citing works
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · see citing works
Transfer Learning in Deep Reinforcement Learning: A Survey
Zhu, Z., Lin, K., Jain, A. K., & Zhou, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292075 · Cited by 861 · see citing works
Overcoming Exploration in Reinforcement Learning with Demonstrations
Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8463162 · Cited by 712 · see citing works
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570 · see citing works
Exploration in deep reinforcement learning: A survey
Ladosz, P., Weng, L., Kim, M., & Oh, H. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.003 · Cited by 525 · see citing works
“(DQfD) [128] differs in two aspects from [127].”
BackgroundMachado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5699 · Cited by 359 · see citing works
Wu, J., Huang, Z., Hu, Z., & Lv, C. (2023) · Engineering · doi:10.1016/j.eng.2022.05.017 · Cited by 205 · see citing works
Adaptive Quantitative Trading: An Imitative Deep Reinforcement Learning Approach
Liu, Y., Liu, Q., Zhao, H., Pan, Z., & Liu, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i02.5587 · Cited by 141 · see citing works
A Survey of Deep Learning Applications to Autonomous Vehicle Control
Kuutti, S., Bowden, R., Jin, Y., Barber, P., & Fallah, S. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2962338 · Cited by 96 · see citing works
Uncertainty-Aware Action Advising for Deep Reinforcement Learning Agents
Da Silva, F. L., Hernandez-Leal, P., Kartal, B., & Taylor, M. E. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6036 · Cited by 69 · see citing works
An Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey
Aubret, A., Matignon, L., & Hassas, S. (2023) · Entropy · doi:10.3390/e25020327 · Cited by 52 · see citing works
Deep reinforcement learning and its applications in medical imaging and radiation therapy: a survey
Xu, L., Zhu, S., & Wen, N. (2022) · Physics in Medicine & Biology · doi:10.1088/1361-6560/ac9cb3 · Cited by 35 · see citing works
Multi-Modal Dynamic Graph Transformer for Visual Grounding
Chen, S., & Li, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01509 · Cited by 28 · see citing works
Kadi, H. A., & Terzić, K. (2023) · Sensors · doi:10.3390/s23052389 · Cited by 19 · see citing works
“Demonstration learning (DemoL) algorithms often used in robotics applications are DQfD [218], DDPGfD [219] and Q-filtered BC [220].”
MethodPlaying the System: Can Puzzle Players Teach us How to Solve Hard Problems?
Mutalova, R., Sarrazin-Gendron, R., Cai, E., Richard, G., Ghasemloo Gheidari, P., Caisse, S., Knight, R., Blanchette, M., Szantner, A., & Waldispühl, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581375 · Cited by 6 · see citing works
- Authors
- Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., Dulac-Arnold, G., Agapiou, J., Leibo, J., & Gruslys, A.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2018-04-29
- Volume
- 32
- Issue
- 1
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2788862220
- Cited by
- 889
- References
- 41
- Citations by year
- 2026: 61 · 2025: 102 · 2024: 118 · 2023: 156 · 2022: 130 · 2021: 119 · 2020: 106 · 2019: 62 · 2018: 30 · 2017: 5
- FWCI
- 32.35
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- Funders
- DeepMind
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- demonstration data, deep reinforcement learning, human demonstrations, deep Q-network, Atari games
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
