Count-Based Exploration with Neural Density Models
Open accessAbstract
Bellemare et al. (2016) introduced the notion of a pseudo-count, derived from a density model, to generalize count-based exploration to non-tabular reinforcement learning. This pseudo-count was used to generate an exploration bonus for a DQN agent and combined with a mixed Monte Carlo update was sufficient to achieve state of the art on the Atari 2600 game Montezuma's Revenge. We consider two questions left open by their work: First, how important is the quality of the density model for exploration? Second, what role does the Monte Carlo update play in exploration? We answer the first question by demonstrating the use of PixelCNN, an advanced neural density model for images, to supply a pseudo-count. In particular, we examine the intrinsic difficulties in adapting Bellemare et al.'s approach when assumptions about the model are violated. The result is a more practical and general algorithm requiring no special apparatus. We combine PixelCNN pseudo-counts with different agent architectures to dramatically improve the state of the art on several hard Atari games. One surprising finding is that the mixed Monte Carlo update is a powerful facilitator of exploration in the sparsest of settings, including Montezuma's Revenge.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
VRB outperforms R3M on all three simulated Franka tasks in behavior cloning with a frozen visual encoder.
“We evaluate on three simulated Franka environments, as shown in Tab. 2, and we see that VRB outperforms R3M on all tasks.”
From Affordances from Human Videos as a Versatile Representation for Robotics (2023) · cites this paper as [84]
Section 4.1 (p. 5) and Table 2 in Appendix D (p. 12) compare exploration agents on Atari games, not VRB and R3M on simulated Franka tasks. Searches for VRB, R3M, Franka, behavior cloning, and a frozen visual encoder found no passage addressing the claimed comparison. (LLM, conf=0.78)
Experimental — accuracy not yet measuredk-NN finds trajectories in the dataset that are close, via distance in feature space, to the goal image.
“k -NN [87] finds trajectories in the dataset that are close (via distance in feature space [84]) to the goal image.”
From Affordances from Human Videos as a Versatile Representation for Robotics (2023) · cites this paper as [84]
The citation to this source is attached to “distance in feature space,” not to the k-NN method. Section 2.2 (p. 2) describes assigning probabilities to images with density models, but neither it nor the additional passages found for feature-space distance describe measuring distance to a goal image in feature space. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318 · see citing works
Flagged first, then by citations
Research works printed but not linked: 28
- Bellemare, Marc, Veness, Joel, and Talvitie, Erik. Skip context tree switching. Proceedings of the International Conference on Machine Learning , 2014.
- Bellemare, Marc G., Naddaf, Yavar, Veness, Joel, and Bowling, Michael. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279, 2013.
- Bellemare, Marc G., Srinivasan, Sriram, Ostrovski, Georg, Schaul, Tom, Saxton, David, and Munos, R´ emi. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems , 2016.
- Deisenroth, Marc P. and Rasmussen, Carl E. PILCO: A model-based and data-efficient approach to policy search. In Proceedings of the International Conference on Machine Learning , 2011.
- French, Robert M. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences , 3(4):128-135, 1999.
- Goodfellow, Ian, Pouget-Abadie, Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair, Sherjil, Courville, Aaron, and Bengio, Yoshua. Generative adversarial nets. In Advances in Neural Information Processing Systems , 2014.
- Gregor, Karol, Danihelka, Ivo, Graves, Alex, Rezende, Danilo, and Wierstra, Daan. Draw: A recurrent neural network for image generation. In Proceedings of the International Conference on Machine Learning , 2015.
- Gruslys, Audrunas, Azar, Mohammad Gheshlaghi, Bellemare, Marc G., and Munos, R´ emi. The Reactor: A sample-efficient actor-critic architecture. arXiv preprint arXiv:1704.04651 , 2017. · arXiv
- Guez, Arthur, Silver, David, and Dayan, Peter. Efficient bayes-adaptive reinforcement learning using samplebased search. In Advances in Neural Information Processing Systems , 2012.
- Houthooft, Rein, Chen, Xi, Duan, Yan, Schulman, John, De Turck, Filip, and Abbeel, Pieter. Variational information maximizing exploration. In Advances in Neural Information Processing Systems (NIPS) , 2016.
- Jaksch, Thomas, Ortner, Ronald, and Auer, Peter. Nearoptimal regret bounds for reinforcement learning. Journal of Machine Learning Research , 11:1563-1600, 2010.
- Kingma, Diederik P. and Welling, Max. Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations , 2013.
- Lopes, Manuel, Lang, Tobias, Toussaint, Marc, and Oudeyer, Pierre-Yves. Exploration in model-based reinforcement learning by empirically estimating learning progress. In Advances in Neural Information Processing Systems , 2012.
- Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, Marc G., Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K., Ostrovski, Georg, et al. Human-level control through deep reinforcement learning. Nature , 518(7540):529533, 2015.
- Munos, R´ emi, Stepleton, Tom, Harutyunyan, Anna, and Bellemare, Marc G. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems , 2016.
- Osband, Ian, van Roy, Benjamin, and Wen, Zheng. Generalization and exploration via randomized value functions. In Proceedings of the International Conference on Machine Learning , 2016.
- Oudeyer, Pierre-Yves, Kaplan, Fr´ ed´ eric, and Hafner, Verena V. Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation , 11(2):265-286, 2007.
- Rezende, Danilo Jimenez, Mohamed, Shakir, and Wierstra, Daan. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of The International Conference on Machine Learning , 2014.
- Strehl, Alexander L. and Littman, Michael L. An analysis of model-based interval estimation for Markov decision processes. Journal of Computer and System Sciences , 74 (8):1309 - 1331, 2008.
- Sutton, Richard S. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Advances in Neural Information Processing Systems , 1996.
- Sutton, Richard S. and Barto, Andrew G. Reinforcement learning: An introduction . MIT Press, 1998.
- Filip, and Abbeel, Pieter. #Exploration: A study of count-based exploration for deep reinforcement learning. arXiv preprint arXiv:1611.04717 , 2016. · arXiv
- Theis, Lucas, van den Oord, A¨ aron, and Bethge, Matthias. A note on the evaluation of generative models. In Proceedings of the International Conference on Learning Representations , 2016.
- Tsitsiklis, John N. and van Roy, Benjamin. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control , 42(5): 674-690, 1997.
- van den Oord, Aaron, Kalchbrenner, Nal, Espeholt, Lasse, Vinyals, Oriol, Graves, Alex, et al. Conditional image generation with PixelCNN decoders. In Advances in Neural Information Processing Systems , 2016a.
- van den Oord, Aaron, Kalchbrenner, Nal, and Kavukcuoglu, Koray. Pixel recurrent neural networks. In Proceedings of the International Conference on Machine Learning , 2016b.
- Veness, Joel, Ng, Kee Siong, Hutter, Marcus, and Bowling, Michael H. Context tree switching. In Proceedings of the Data Compression Conference , 2012.
- Wang, Ziyu, Schaul, Tom, Hessel, Matteo, van Hasselt, Hado, Lanctot, Marc, and de Freitas, Nando. Dueling network architectures for deep reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning , pp. 1995-2003, 2016.
Cited by
Sorted by citations
Deep Q-learning From Demonstrations
Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., et al. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11757 · Cited by 889 · see citing works
Wang, X., Huang, Q., Celikyilmaz, A., Gao, J., Shen, D., Wang, Y.-F., Wang, W. Y., & Zhang, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00679 · Cited by 522 · see citing works
Exploration in deep reinforcement learning: A survey
Ladosz, P., Weng, L., Kim, M., & Oh, H. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.003 · Cited by 525 · see citing works
Exploration by Random Network Distillation
Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018) · arXiv · doi:10.48550/arxiv.1810.12894 · Cited by 527 · see citing works
Emergent Tool Use From Multi-Agent Autocurricula
Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019) · arXiv · doi:10.48550/arxiv.1909.07528 · Cited by 424 · see citing works
“One example is count-based exploration, where agents are incentivized to reach infrequently visited states by maintaining state visitation counts [37, 38, 39] or density estimators [40, 41].”
BackgroundMachado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5699 · Cited by 359 · see citing works
Self-citationExploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
Hao, J., Yang, T., Tang, H., Bai, C., Liu, J., Meng, Z., Liu, P., & Wang, Z. (2021) · arXiv · doi:10.48550/arxiv.2109.06668 · Cited by 192
Planning to Explore via Self-Supervised World Models
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., & Pathak, D. (2020) · arXiv · doi:10.48550/arxiv.2005.05960 · Cited by 129 · see citing works
Affordances from Human Videos as a Versatile Representation for Robotics
Bahl, S., Mendonca, R., Chen, L., Jain, U., & Pathak, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01324 · Cited by 119 · see citing works
“Exploration methods are usually guided by intrinsic rewards that are selfgenerated by the robotic agent, and are not specific to any task [9, 47, 49, 60, 69, 81, 85, 88, 93, 110].”
BackgroundRule-Based Reinforcement Learning for Efficient Robot Navigation With Space Reduction
Zhu, Y., Wang, Z., Chen, C., & Dong, D. (2022) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2021.3072675 · Cited by 57 · see citing works
An Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey
Aubret, A., Matignon, L., & Hassas, S. (2023) · Entropy · doi:10.3390/e25020327 · Cited by 52 · see citing works
Count-based exploration with the successor representation
Machado, M. C., Bellemare, M. G., & Bowling, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5955 · Cited by 19 · see citing works
Self-citationZhou, Q., Li, H., & Wang, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6177 · Cited by 12 · see citing works
Playtesting: What is Beyond Personas
Ariyurek, S., Surer, E., & Betin-Can, A. (2023) · IEEE Transactions on Games · doi:10.1109/tg.2022.3165882 · Cited by 7 · see citing works
“The count is formulated using a density model [12], a neural density model [22], a hash table [23], and exemplar models [13].”
MethodA Study of Global and Episodic Bonuses for Exploration in Contextual MDPs
Henaff, M., Jiang, M., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2306.03236 · Cited by 1 · see citing works
“…as RND (Burda et al., 2019b), ICM (Pathak et al., 2017a) and pseudocounts (Bellemare et al., 2016b; Strehl & Littman, 2008; Bellemare et al., 2016a; Ostrovski et al., 2017; Martin et al., 2017; Tang et al., 2017; Machado et al., 2020), as well as model-based approaches (Shyam et al., 2019; Henaff,…”
BackgroundImproving Environment Novelty Quantification for Effective Unsupervised Environment Design
Teoh, J., Li, W., & Varakantham, P. (2025) · arXiv
“Notably, density models have been flexibly used to model state-action visitations for both large discrete state-action spaces (via pseudo-counts [8, 36]) and continuous state-action spaces [62, 61].”
Some Considerations on Learning to Explore via Meta-Reinforcement Learning
Stadie, B. C., Yang, G., Houthooft, R., Chen, X., Duan, Y., Wu, Y., Abbeel, P., & Sutskever, I. (2018) · arXiv · doi:10.48550/arxiv.1803.01118 · Cited by 74 · see citing works
- Authors
- Ostrovski, G., Bellemare, M. G., Oord, A. V. D., & Munos, R.
- ORCID
- Bellemare, Marc G.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- Pages
- 2721-2730
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1703.01310
- OpenAlex
- W2596982695
- Cited by
- 377
- References
- 1
- Citations by year
- 2026: 1 · 2025: 11 · 2024: 24 · 2023: 48 · 2022: 38 · 2021: 79 · 2020: 76 · 2019: 61 · 2018: 32 · 2017: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Keywords
- pseudo counts, PixelCNN, exploration bonus, sparse rewards, deep Q-network, Atari games, Montezuma's Revenge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
