Exploration by Random Network Distillation
Open accessAbstract
We introduce an exploration bonus for deep reinforcement learning methods that is easy to implement and adds minimal overhead to the computation performed. The bonus is the error of a neural network predicting features of the observations given by a fixed randomly initialized neural network. We also introduce a method to flexibly combine intrinsic and extrinsic rewards. We find that the random network distillation (RND) bonus combined with this increased flexibility enables significant progress on several hard exploration Atari games. In particular we establish state of the art performance on Montezuma's Revenge, a game famously difficult for deep reinforcement learning methods. To the best of our knowledge, this is the first method that achieves better than average human performance on this game without using demonstrations or having access to the underlying state of the game, and occasionally completes the first level.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Solving detachment and derailment requires a good global exploration strategy that can better balance long-term and short-term environment information.
“Solving these problems requires a good global exploration strategy [211] that can better balance long-term and short-term environment information [195].”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [211]
The Discussion (p. 10) says RND handles local exploration but cannot manage global exploration requiring coordinated decisions over long time horizons, and calls solving such problems an important direction for future work. This supports the need for better global exploration, but the paper does not identify detachment and derailment as the problems it would solve. The claim about balancing long- and short-term information is attached to a different citation. (LLM, conf=0.83)
Experimental — accuracy not yet measuredThe current approach is vulnerable to the noisy TV problem, where novelty-driven level prioritization may focus on unpredictable noise elements of the environment rather than beneficial learning experiences.
“Specifically, although our experiments showed strong empirical gains in simplified environments, the current approach is vulnerable to the noisy TV problem [19], where novelty-driven level prioritization may focus on unpredictable noise elements of the environment, rather than beneficial learning experiences.”
From Improving Environment Novelty Quantification for Effective Unsupervised Environment Design (2024) · cites this paper as [19]
Section 2.2.1 (p. 3) explains that prediction-error exploration can be drawn to unpredictable environmental noise—the noisy-TV problem. But the paper describes this vulnerability for forward-dynamics prediction, not for the citing authors’ novelty-driven level prioritization; its Introduction (p. 2) presents random network distillation as a way to avoid that source of error. (LLM, conf=0.86)
Experimental — accuracy not yet measuredThe cap solution limits the agent looping over distinct states or getting stuck in a state like the noisy TV problem.
“This solution limits the agent looping over distinct states or getting stuck in a state like the noisy TV problem [25].”
From Playtesting: What is Beyond Personas (2023) · cites this paper as [25]
The Introduction (p. 2) presents random network distillation as a way to address the noisy-TV problem. The comparison to baselines (p. 9) reports a dynamics-based agent oscillating between two rooms because of that problem. The paper does not identify a cap solution or show that capping prevents an agent from looping or getting stuck. (LLM, conf=0.83)
Experimental — accuracy not yet measuredSince the feedback is infinite, the agent may loop over novel states or get stuck in a novel state.
“Since the feedback is infinite, the agent may loop over novel states or get stuck in a novel state [25].”
From Playtesting: What is Beyond Personas (2023) · cites this paper as [25]
Section 3.7 (p. 9) reports that the RND agent repeatedly moved back and forth around dangerous, rarely experienced states, supporting the possibility of a loop. Section 2.3 (p. 4) describes a non-episodic intrinsic return, but does not call the feedback infinite or say that the agent gets stuck in a single novel state. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · see citing works
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
Self-citationPlaying Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471 · see citing works
Neural Architecture Search with Reinforcement LearningPreprint
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 5,091 · see citing works
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317 · see citing works
Random Features for Large-Scale Kernel Machines
Rahimi, A., & Recht, B. (2007) · Cited by 2,783 · see citing works
What is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153 · see citing works
Rainbow: Combining Improvements in Deep Reinforcement Learning
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., & Silver, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11796 · Cited by 1,846 · see citing works
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815 · see citing works
Intrinsic Motivation Systems for Autonomous Mental Development
Oudeyer, P.-Y., Kaplan, F., & Hafner, V. V. (2007) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2006.890271 · Cited by 1,126 · see citing works
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic MotivationPreprint
Kulkarni, T. D., Narasimhan, K., Saeedi, A., & Tenenbaum, J. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06057 · Cited by 874 · see citing works
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner ArchitecturesPreprint
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.01561 · Cited by 851 · see citing works
Deep Exploration via Bootstrapped DQNPreprint
Osband, I., Blundell, C., Pritzel, A., & Roy, B. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.04621 · Cited by 837 · see citing works
Curious model-building control systems
Schmidhuber, J. (1991) · [Proceedings] 1991 IEEE International Joint Conference on Neural Networks · doi:10.1109/ijcnn.1991.170605 · Cited by 733 · see citing works
A Distributional Perspective on Reinforcement LearningPreprint
Bellemare, M. G., Dabney, W., & Munos, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06887 · Cited by 724 · see citing works
A possibility for implementing curiosity and boredom in model-building neural controllers
Schmidhuber, J. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/3115.003.0030 · Cited by 669 · see citing works
Noisy Networks for ExplorationPreprint
Fortunato, M., Azar, M. G., Piot, B., Menick, J., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.10295 · Cited by 664 · see citing works
Distributed Prioritized Experience ReplayPreprint
Horgan, D., Quan, J., Budden, D. M., Barth-Maron, G., Hessel, M., van Hasselt, H. P., & Silver, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.00933 · Cited by 557 · see citing works
Incentivizing Exploration In Reinforcement Learning With Deep Predictive ModelsPreprint
Stadie, B. C., Levine, S., & Abbeel, P. (2015) · arXiv.org · doi:10.48550/arxiv.1507.00814 · Cited by 545 · see citing works
#Exploration: A Study of Count-Based Exploration for Deep Reinforcement LearningPreprint
Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04717 · Cited by 486 · see citing works
An analysis of model-based Interval Estimation for Markov Decision Processes
Strehl, A. L., & Littman, M. L. (2008) · Journal of Computer and System Sciences · doi:10.1016/j.jcss.2007.08.009 · Cited by 428 · see citing works
Parameter Space Noise for ExplorationPreprint
Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R. Y., Chen, X., Asfour, T., Abbeel, P., & Andrychowicz, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.01905 · Cited by 423 · see citing works
Large-Scale Study of Curiosity-Driven Learning
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., & Efros, A. A. (2018) · arXiv · doi:10.48550/arxiv.1808.04355 · Cited by 401 · see citing works
Self-citationCount-Based Exploration with Neural Density Models
Ostrovski, G., Bellemare, M. G., Oord, A. V. D., & Munos, R. (2017) · arXiv · doi:10.48550/arxiv.1703.01310 · Cited by 377 · see citing works
Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5699 · Cited by 359 · see citing works
On Random Weights and Unsupervised Feature Learning
Saxe, A., Koh, P. W., Chen, Z., Bhand, M., Suresh, B., & Ng, A. Y. (2011) · Cited by 353 · see citing works
Unifying Count-Based Exploration and Intrinsic Motivation
Bellemare, M. G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., & Munos, R. (2016) · arXiv · doi:10.48550/arxiv.1606.01868 · Cited by 264 · see citing works
An information-theoretic approach to curiosity-driven reinforcement learning
Still, S., & Precup, D. (2012) · Theory in Biosciences · doi:10.1007/s12064-011-0142-z · Cited by 165 · cited 1× · see citing works
“This has motivated work on exploration via quantification of uncertainty (Still & Precup, 2012; Houthooft et al., 2016) or prediction improvement measures (Schmidhuber, 1991a; Oudeyer et al., 2007; Lopes et al., 2012; Achiam & Sastry, 2017).”
MethodTemporal Difference Models: Model-Free Deep RL for Model-Based ControlPreprint
Pong, V. H., Gu, S., Dalal, M., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09081 · Cited by 164 · see citing works
Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
Achiam, J., & Sastry, S. (2017) · arXiv · doi:10.48550/arxiv.1703.01732 · Cited by 139 · see citing works
Randomized Prior Functions for Deep Reinforcement LearningPreprint
Osband, I., Aslanides, J., & Cassirer, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.03335 · Cited by 124 · see citing works
Variational Option Discovery AlgorithmsPreprint
Achiam, J., Edwards, H. S., Amodei, D., & Abbeel, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.10299 · Cited by 82 · see citing works
Self-citationObserve and Look Further: Achieving Consistent Performance on AtariPreprint
Pohlen, T., Piot, B., Hester, T., Azar, M. G., Horgan, D., Budden, D. M., Barth-Maron, G., van Hasselt, H. P., Quan, J., Vecerík, M., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11593 · Cited by 81 · see citing works
UCB Exploration via Q-EnsemblesPreprint
Chen, R. Y., Sidor, S., Abbeel, P., & Schulman, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.01502 · Cited by 74 · see citing works
Intrinsic motivation and automatic curricula via asymmetric self-play
Sukhbaatar, S., Lin, Z., Kostrikov, I., Synnaeve, G., Szlam, A. D., & Fergus, R. (2018) · International Conference on Learning Representations · Cited by 72 · see citing works
Learning to Perform Physics Experiments via Deep Reinforcement LearningPreprint
Denil, M., Agrawal, P., Kulkarni, T. D., Erez, T., Battaglia, P., & de Freitas, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01843 · Cited by 58 · see citing works
The Uncertainty Bellman Equation and ExplorationPreprint
O’Donoghue, B., Osband, I., Munos, R., & Mnih, V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.05380 · Cited by 53 · see citing works
Stanton, C., & Clune, J. (2018) · arXiv.org · doi:10.48550/arxiv.1806.00553 · Cited by 42 · see citing works
DORA The Explorer: Directed Outreaching Reinforcement Action-SelectionPreprint
Choshen, L., Fox, L., & Loewenstein, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.04012 · Cited by 35 · see citing works
Count-based exploration with the successor representation
Machado, M. C., Bellemare, M. G., & Bowling, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5955 · Cited by 19 · see citing works
Expert-augmented actor-critic for ViZDoom and Montezumas RevengePreprint
Garmulewicz, M., Michalewski, H., & Miłoś, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03447 · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 13
- Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching YouTube. arXiv preprint arXiv:1805.11592 , 2018. · arXiv
- Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint , 2018.
- Justin Fu, John D Co-Reyes, and Sergey Levine. EX2: Exploration with exemplar models for deep reinforcement learning. NIPS , 2017.
- Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. ICLR Workshop , 2017.
- Nick Haber, Damian Mrowca, Li Fei-Fei, and Daniel LK Yamins. Learning to play with intrinsicallymotivated self-aware agents. arXiv preprint arXiv:1802.07442 , 2018. · arXiv
- Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: Variational information maximizing exploration. In NIPS , 2016.
- Manuel Lopes, Tobias Lang, Marc Toussaint, and Pierre-Yves Oudeyer. Exploration in model-based reinforcement learning by empirically estimating learning progress. In NIPS , 2012.
- Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, February 2015.
- Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML , 2016.
- Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. arXiv preprint arXiv:1806.05635 , 2018. · arXiv
- OpenAI, :, M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba. Learning Dexterous In-Hand Manipulation. ArXiv e-prints , August 2018.
- Georg Ostrovski, Marc G Bellemare, Aaron van den Oord, and R´ emi Munos. Count-based exploration with neural density models. International Conference for Machine Learning , 2018.
- Zichao Yang, Marcin Moczulski, Misha Denil, Nando de Freitas, Alex Smola, Le Song, and Ziyu Wang. Deep fried convnets. In Proceedings of the IEEE International Conference on Computer Vision , pp. 1476-1483, 2015.
Cited by
Sorted by citations
Dota 2 with Large Scale Deep Reinforcement Learning
OpenAI, :, Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.06680 · Cited by 1,065 · see citing works
Conservative Q-Learning for Offline Reinforcement LearningPreprint
Kumar, A., Zhou, A., Tucker, G., & Levine, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04779 · Cited by 776 · see citing works
Exploration in deep reinforcement learning: A survey
Ladosz, P., Weng, L., Kim, M., & Oh, H. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.003 · Cited by 525 · see citing works
“Stanton and Clune [51] agent position, reward grid A2C Prediction error Atari: Montezuma Revenge 3200 (A2C 0) Atari images Ac MB D/ D Achiam and Sastry [33] TRPO Prediction error Mujoco: halfcheetah 80 (VIME 40); Atari: Venture 400 (VIME 0) Atari RAM states/ Mujoco joints angles Ac MB C/ C Li et al. [44] A2C Predictio…”
BackgroundInfluentialA Survey of Embodied AI: From Simulators to Research Tasks
Duan, J., Yu, S., Tan, H. L., Zhu, H., & Tan, C. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3141105 · Cited by 420 · see citing works
Computational mechanisms of curiosity and goal-directed exploration
Schwartenbeck, P., Passecker, J., Hauser, T. U., FitzGerald, T. H. B., Kronbichler, M., & Friston, K. J. (2019) · eLife · doi:10.7554/elife.41703 · Cited by 250 · see citing works
Agent57: Outperforming the Atari Human BenchmarkPreprint
Badia, A. P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, D., & Blundell, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.13350 · Cited by 216 · see citing works
Episodic Curiosity through ReachabilityPreprint
Savinov, N., Raichuk, A., Marinier, R., Vincent, D., Pollefeys, M., Lillicrap, T., & Gelly, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.02274 · Cited by 231 · see citing works
Go-Explore: a New Approach for Hard-Exploration Problems
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J. (2019) · arXiv · doi:10.48550/arxiv.1901.10995 · Cited by 205 · see citing works
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J. (2021) · RePEc: Research Papers in Economics · Cited by 201 · see citing works
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation
Gadre, S. Y., Wortsman, M., Ilharco, G., Schmidt, L., & Song, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02219 · Cited by 192 · see citing works
Never Give Up: Learning Directed Exploration Strategies
Badia, A. P., Sprechmann, P., Vitvitskyi, A., Guo, D., Piot, B., Kapturowski, S., Tieleman, O., Arjovsky, M., Pritzel, A., Bolt, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.06038 · Cited by 170 · see citing works
Benchmarking Batch Deep Reinforcement Learning AlgorithmsPreprint
Fujimoto, S., Conti, E., Ghavamzadeh, M., & Pineau, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.01708 · Cited by 166 · see citing works
(2021) · UCL Discovery (University College London) · Cited by 141 · see citing works
Planning to Explore via Self-Supervised World Models
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., & Pathak, D. (2020) · arXiv · doi:10.48550/arxiv.2005.05960 · Cited by 129 · see citing works
RoboNet: Large-Scale Multi-Robot Learning
Dasari, S., Ebert, F., Tian, S., Nair, S., Bucher, B., Schmeckpeper, K., Singh, S., Levine, S., & Finn, C. (2019) · Digital Eprints Services at ISB (DESI) (Indian School of Business) · doi:10.48550/arxiv.1910.11215 · Cited by 151 · see citing works
Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey
Majid, A. Y., Majid, A. Y., Saaybi, S., van Rietbergen, T., François-Lavet, V., Prasad, R. R. V., & Verhoeven, C. J. M. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3264540 · Cited by 108 · see citing works
An Exploration of Embodied Visual Exploration
Ramakrishnan, S. K., Jayaraman, D., & Grauman, K. L. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01437-z · Cited by 101 · see citing works
The Ingredients of Real-World Robotic Reinforcement LearningPreprint
Zhu, H., Yu, J., Gupta, A., Shah, D., Hartikainen, K., Singh, A., Kumar, V., & Levine, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12570 · Cited by 109 · see citing works
Efficient Exploration via State Marginal MatchingPreprint
Lee, L., Eysenbach, B., Parisotto, E., Xing, E. P., Levine, S., & Salakhutdinov, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05274 · Cited by 97 · see citing works
Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning
Juliani, A., Khalifa, A., Berges, V.-P., Harper, J., Teng, E., Henry, H., Crespi, A., Togelius, J., & Lange, D. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/373 · Cited by 93 · see citing works
Skew-Fit: State-Covering Self-Supervised Reinforcement LearningPreprint
Pong, V. H., Dalal, M., Lin, S., Nair, A., Bahl, S., & Levine, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.03698 · Cited by 90 · see citing works
A survey on intrinsic motivation in reinforcement learningPreprint
Aubret, A., Matignon, L., & Hassas, S. (2019) · arXiv.org · doi:10.48550/arxiv.1908.06976 · Cited by 175 · see citing works
Deep Reinforcement Learning on Autonomous Driving Policy With Auxiliary Critic Network
Wu, Y., Liao, S., Liu, X., Li, Z., & Lu, R. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3116063 · Cited by 86 · see citing works
Colas, C., Karch, T., Sigaud, O., & Oudeyer, P. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13554 · Cited by 101 · see citing works
Wang, H., Liu, C. H., Yang, H., Wang, G., & Leung, K. K. (2023) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2023.3289172 · Cited by 75 · see citing works
Gupta, A., Yu, J., Zhao, T. Z., Kumar, V., Rovinsky, A., Xu, K., Devlin, T. J., & Levine, S. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561384 · Cited by 75 · see citing works
Reinforcement Learning with Prototypical RepresentationsPreprint
Yarats, D., Fergus, R., Lazaric, A., & Pinto, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.11271 · Cited by 64 · see citing works
Celebrating Diversity in Shared Multi-Agent Reinforcement LearningPreprint
Li, C., Wang, T., Wu, C., Zhao, Q., Yang, J. J., & Zhang, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02195 · Cited by 64 · see citing works
Deep Reinforcement Learning: A State-of-the-Art Walkthrough
Lazaridis, A., Fachantidis, A., & Vlahavas, I. P. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12412 · Cited by 58 · see citing works
Sun, P., Zhou, W., & Li, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6049 · Cited by 54 · see citing works
Visual novelty, curiosity, and intrinsic reward in machine learning and the brain
Jaegle, A., Mehrpour, V., & Rust, N. C. (2019) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2019.08.004 · Cited by 54 · see citing works
BND*-DDQN: Learn to Steer Autonomously Through Deep Reinforcement Learning
Wu, K., Wang, H., Esfahani, M. A., & Yuan, S. (2019) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2019.2928820 · Cited by 54 · see citing works
Real-Time Object Navigation With Deep Neural Networks and Hierarchical Reinforcement Learning
Staroverov, A., Yudin, D. A., Belkin, I., Adeshkin, V., Solomentsev, Y., & Panov, A. I. (2020) · IEEE Access · doi:10.1109/access.2020.3034524 · Cited by 53 · see citing works
An Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey
Aubret, A., Matignon, L., & Hassas, S. (2023) · Entropy · doi:10.3390/e25020327 · Cited by 52 · see citing works
Hyperbolic Discounting and Learning over Multiple HorizonsPreprint
Fedus, W., Gelada, C., Bengio, Y., Bellemare, M. G., & Larochelle, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06865 · Cited by 52 · see citing works
PLGRIM: Hierarchical Value Learning for Large-scale Exploration in Unknown Environments
Kim, S.-K., Bouman, A., Salhotra, G., Fan, D. D., Otsu, K., Burdick, J. W., & Agha–mohammadi, A. (2021) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v31i1.16014 · Cited by 49 · see citing works
How much intelligence is there in artificial intelligence? A 2020 update
van der Maas, H. L. J., Snoek, L., & Stevenson, C. E. (2021) · Intelligence · doi:10.1016/j.intell.2021.101548 · Cited by 50 · see citing works
Dealing with Sparse Rewards in Reinforcement LearningPreprint
Hare, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.09281 · Cited by 71 · see citing works
Subramanian, J., Sinha, A. K., Seraj, R., & Mahajan, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.08843 · Cited by 44 · see citing works
Distributed Deep Reinforcement Learning: A Survey and a Multi-player Multi-agent Learning Toolbox
Yin, Q., Yu, T., Shen, S., Yang, J. J., Zhao, M., Ni, W., Huang, K., Liang, B., & Wang, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-023-1454-4 · Cited by 45 · see citing works
Meta-learning curiosity algorithmsPreprint
Alet, F., Schneider, M. F., Lozano‐Pérez, T., & Kaelbling, L. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.05325 · Cited by 45 · see citing works
Hu, B., & Li, J. (2020) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2020.3007100 · Cited by 43 · see citing works
Diversity policy gradient for sample efficient quality-diversity optimization
Pierrot, T., Macé, V., Chalumeau, F., Flajolet, A., Cideron, G., Beguir, K., Cully, A., Sigaud, O., & Perrin, N. (2022) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3512290.3528845 · Cited by 62 · see citing works
Information Theoretic Regret Bounds for Online Nonlinear ControlPreprint
Kakade, S. M., Krishnamurthy, A., Lowrey, K., Ohnishi, M., & Sun, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12466 · Cited by 41 · see citing works
Transformation-based Adversarial Video Prediction on Large-Scale DataPreprint
Luc, P., Clark, A., Dieleman, S., Casas, D. D. L., Doron, Y., Cassirer, A., & Simonyan, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04035 · Cited by 53 · see citing works
Games for Artificial Intelligence Research: A Review and Perspectives
Hu, C., Zhao, Y., Wang, Z., Du, H., & Liu, J. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3410935 · Cited by 51 · see citing works
Teaching on a Budget in Multi-Agent Deep Reinforcement Learning
İlhan, E., Gow, J., & Pérez-Liébana, D. (2019) · IEEE Conference on Games (CoG) · doi:10.1109/cig.2019.8847988 · Cited by 36 · see citing works
Li, J., Shi, H., & Hwang, K. (2023) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2023.3323307 · Cited by 35 · see citing works
Reinforcement learning for robotic manipulation using simulated locomotion demonstrations
Kilinc, O., & Montana, G. (2021) · Machine Learning · doi:10.1007/s10994-021-06116-1 · Cited by 37 · see citing works
Deep reinforcement learning and its applications in medical imaging and radiation therapy: a survey
Xu, L., Zhu, S., & Wen, N. (2022) · Physics in Medicine & Biology · doi:10.1088/1361-6560/ac9cb3 · Cited by 35 · see citing works
Toward Semantic Communication Protocols: A Probabilistic Logic Perspective
Seo, S., Park, J., Ko, S., Choi, J., Bennis, M., & Kim, S. (2023) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2023.3288268 · Cited by 39 · see citing works
Learning Agile Locomotion on Risky Terrains
Zhang, C., Rudin, N., Hoeller, D., & Hutter, M. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801909 · Cited by 37 · see citing works
Lee, G. T., & Kim, C. O. (2020) · IEEE Access · doi:10.1109/access.2020.3046284 · Cited by 34 · see citing works
Contrastive Identity-Aware Learning for Multi-Agent Value Decomposition
Liu, S., Zhou, Y., Song, J., Zheng, T., Chen, K., Zhu, T., Feng, Z., & Song, M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i10.26370 · Cited by 32 · see citing works
Liu, C. H., Zhao, Y., Dai, Z., Yuan, Y., Wang, G., Wu, D. O., & Leung, K. K. (2020) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde48307.2020.00010 · Cited by 32 · see citing works
Fast and slow curiosity for high-level exploration in reinforcement learning
Bougie, N., & Ichise, R. (2020) · Applied Intelligence · doi:10.1007/s10489-020-01849-3 · Cited by 33 · see citing works
Curious Representation Learning for Embodied Intelligence
Du, Y., Gan, C., & Isola, P. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01024 · Cited by 32 · see citing works
Skill-based curiosity for intrinsically motivated reinforcement learning
Bougie, N., & Ichise, R. (2019) · Machine Learning · doi:10.1007/s10994-019-05845-8 · Cited by 31 · see citing works
Curiosity-Driven Exploration via Latent Bayesian Surprise
Mazzaglia, P., Çatal, O., Verbelen, T., & Dhoedt, B. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20743 · Cited by 31 · see citing works
Action and Perception as Divergence MinimizationPreprint
Hafner, D., Ortega, P. A., Ba, J. L., Parr, T., Friston, K. J., & Heess, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.01791 · Cited by 32 · see citing works
Improved Learning of Robot Manipulation Tasks Via Tactile Intrinsic Motivation
Vulin, N., Christen, S., Stevšić, S., & Hilliges, O. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3061308 · Cited by 29 · see citing works
Explorer-Actor-Critic: Better actors for deep reinforcement learning
Zhang, J., Han, S. D., Xiong, X., Zhu, S., & Lü, S. (2024) · Information Sciences · doi:10.1016/j.ins.2024.120255 · Cited by 28 · see citing works
Kamuni, N., Shah, H., Chintala, S., Kunchakuri, N., & Alla, S. (2024) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc59802.2024.00063 · Cited by 29 · see citing works
Reinforcement Learning with Videos: Combining Offline Observations with InteractionPreprint
Schmeckpeper, K., Rybkin, O., Daniilidis, K., Levine, S., & Finn, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.06507 · Cited by 32 · see citing works
Uncertainty-aware Self-supervised Target-mass Grasping of Granular Foods
Takahashi, K., Ko, W., Ummadisingu, A., & Maeda, S. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561728 · Cited by 27 · see citing works
Automated Gameplay Testing and Validation With Curiosity-Conditioned Proximal Trajectories
Sestini, A., Gisslén, L., Bergdahl, J., Tollmar, K., & Bagdanov, A. D. (2022) · IEEE Transactions on Games · doi:10.1109/tg.2022.3226910 · Cited by 33 · see citing works
Exploration Based Language Learning for Text-Based Games
Madotto, A., Namazifar, M., Huizinga, J., Molino, P., Ecoffet, A., Zheng, H., Papangelis, A., Yu, D., Khatri, C., & Tür, G. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/207 · Cited by 27 · see citing works
Robust Proactive Power Smoothing Control of PV Systems Based on Deep Reinforcement Learning
Chen, X., Xu, X., Wang, J., Fang, L., Du, Y., Lim, E. G., & Ma, J. (2023) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2023.3239852 · Cited by 25 · see citing works
Machine Learning With Computer Networks: Techniques, Datasets, and Models
Afifi, H., Pochaba, S., Boltres, A., Laniewski, D., Haberer, J., Paeleke, L., Poorzare, R., Stolpmann, D., Wehner, N., Redder, A., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3384460 · Cited by 32 · see citing works
Offline Reinforcement Learning as Anti-exploration
Rezaeifar, S., Dadashi, R., Vieillard, N., Hussenot, L., Bachem, O., Pietquin, O., & Geist, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20783 · Cited by 25 · see citing works
Chen, Z., Silvestri, F., Tolomei, G., Wang, J., Zhu, H., & Ahn, H. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3223892 · Cited by 25 · see citing works
ACDER: Augmented Curiosity-Driven Experience Replay
Li, B., Lü, T., Li, J., Lu, N., Cai, Y., & Wang, S. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197421 · Cited by 24 · see citing works
Osa, T., Tangkaratt, V., & Sugiyama, M. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.04.009 · Cited by 24 · see citing works
Comprehensible Context-driven Text Game Playing
Yin, X., & May, J. (2019) · 2019 IEEE Conference on Games (CoG) · doi:10.1109/cig.2019.8847954 · Cited by 33 · see citing works
Improving robot navigation in crowded environments using intrinsic rewards
Martínez, D., Riazuelo, L., & Montano, L. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160876 · Cited by 22 · see citing works
Nguyen, T., Luu, T. M., Vu, T., & Yoo, C. D. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636536 · Cited by 22 · see citing works
Focus on Impact: Indoor Exploration With Intrinsic Motivation
Bigazzi, R., Landi, F., Cascianelli, S., Baraldi, L., Cornia, M., & Cucchiara, R. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3145971 · Cited by 22 · see citing works
Diversity-augmented intrinsic motivation for deep reinforcement learning
Dai, T., Du, Y., Fang, M., & Bharath, A. A. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.10.040 · Cited by 21 · see citing works
Summerfield, C. (2022) · doi:10.1093/oso/9780192843883.001.0001 · Cited by 23 · see citing works
Reinforcement Learning for Branch-and-Bound Optimisation Using Retrospective Trajectories
Parsonson, C. W. F., Laterre, A., & Barrett, T. D. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25521 · Cited by 22 · see citing works
Exploration by Maximizing Renyi Entropy for Reward-Free RL Framework
Zhang, C., Cai, Y., Huang, L., & Li, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17297 · Cited by 24 · see citing works
Self-Attention-Based Temporary Curiosity in Reinforcement Learning Exploration
Hu, H., Song, S., & Huang, G. M. (2019) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2019.2957051 · Cited by 21 · see citing works
Ocana, J. M. C., Capobianco, R., & Nardi, D. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14390 · Cited by 21 · see citing works
Generative Model-Based Testing on Decision-Making Policies
Li, Z., Wu, X., Zhu, D., Cheng, M., Chen, S., Zhang, F., Xie, X., Ma, L., & Zhao, J. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00153 · Cited by 21 · see citing works
Matheron, G., Perrin, N., & Sigaud, O. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-61616-8_24 · Cited by 21 · see citing works
Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
Bai, C., Liu, P., Liu, K., Wang, L., Zhao, Y., Han, L., & Wang, Z. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3129160 · Cited by 21 · see citing works
Novelty Search in representational space for sample efficient explorationPreprint
Tao, R.-Y., François-Lavet, V., & Pineau, J. (2020) · Neural Information Processing Systems · doi:10.48550/arxiv.2009.13579 · Cited by 54 · see citing works
Conservative Uncertainty Estimation By Fitting Prior Networks
Ciosek, K., Fortuin, V., Tomioka, R., Hofmann, K., & Turner, R. E. (2020) · International Conference on Learning Representations · Cited by 20 · see citing works
Zhang, T., Zhu, K., Wang, J., & Han, Z. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2021.3120266 · Cited by 20 · see citing works
LanCon-Learn: Learning With Language to Enable Generalization in Multi-Task Manipulation
Silva, A., Moorman, N. M., Silva, W., Zaidi, Z., Gopalan, N., & Gombolay, M. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3139667 · Cited by 20 · see citing works
Optimistic Exploration even with a Pessimistic InitialisationPreprint
Rashid, T., Peng, B., Böhmer, W., & Whiteson, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12174 · Cited by 20 · see citing works
Count-based exploration with the successor representation
Machado, M. C., Bellemare, M. G., & Bowling, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5955 · Cited by 19 · see citing works
Curiosity-Driven Exploration for Off-Policy Reinforcement Learning Methods
Li, B., Lü, T., Li, J., Lu, N., Cai, Y., & Wang, S. (2019) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio49542.2019.8961529 · Cited by 19 · see citing works
Batch Exploration With Examples for Scalable Robotic Reinforcement Learning
Chen, A. S., Nam, H., Nair, S., & Finn, C. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3068655 · Cited by 21 · see citing works
Worst-Case Regret Bounds for Exploration via Randomized Value FunctionsPreprint
Russo, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02870 · Cited by 19 · see citing works
LiDAR-Based End-to-End Active SLAM Using Deep Reinforcement Learning in Large-Scale Environments
Chen, J., Wu, K., Hu, M., Suganthan, P. N., & Makur, A. (2024) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2024.3405483 · Cited by 18 · see citing works
DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical RepresentationsPreprint
Deng, F., Jang, I., & Ahn, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.14565 · Cited by 18 · see citing works
Unified curiosity-Driven learning with smoothed intrinsic reward estimation
Huang, F., Li, W., Cui, J., Fu, Y., & Li, X. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108352 · Cited by 17 · see citing works
Zhang, S., Liu, S., Sun, J., Chen, Y., Huang, W., Liu, J., Liu, J., & Hao, J. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678832 · Cited by 17 · see citing works
Unsupervised Reinforcement Learning for Transferable Manipulation Skill Discovery
Cho, D., Kim, J.-G., & Kim, H. J. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3171915 · Cited by 17 · see citing works
- Authors
- Burda, Y., Edwards, H., Storkey, A., & Klimov, O.
- Affiliations
- OpenAI (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1810.12894
- OpenAlex
- W2899205164
- Cited by
- 527
- References
- 41
- Citations by year
- 2026: 2 · 2025: 35 · 2024: 75 · 2023: 77 · 2022: 79 · 2021: 115 · 2020: 87 · 2019: 52 · 2018: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Keywords
- random network distillation, exploration bonus, deep reinforcement learning, intrinsic rewards, Atari games, Montezuma's Revenge, feature prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
