CoChat
Sign Up
PreprintCited by 95Reinforcement Learning in RoboticsArtificial Intelligence in GamesAdvanced Bandit Algorithms Research

EX2: Exploration with Exemplar Models for Deep Reinforcement Learning

Open access
Fu, Justin, Co-Reyes, John D., Levine, Sergey·2017·arXiv
DOI 10.48550/arxiv.1703.01260arXiv 1703.01260
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep reinforcement learning algorithms have been shown to learn complex tasks using highly general policy classes. However, sparse reward problems remain a significant challenge. Exploration methods based on novelty detection have been particularly successful in such settings but typically require generative or predictive models of the observations, which can be difficult to train when the observations are very high-dimensional and complex, as in the case of raw images. We propose a novelty detection algorithm for exploration that is based entirely on discriminatively trained exemplar models, where classifiers are trained to discriminate each visited state against all others. Intuitively, novel states are easier to distinguish against other states seen during training. We show that this kind of discriminative modeling corresponds to implicit density estimation, and that it can be combined with count-based exploration to produce competitive results on a range of popular benchmark tasks, including state-of-the-art results on challenging egocentric observations in the vizDoom benchmark.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1703.01260

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Modern exploration algorithms entice exploration logically, unlike traditional epsilon-greedy where exploration is achieved through randomness.

    “Compared to the traditional exploration methods such as ϵ -greedy, where exploration is achieved through randomness, these modern algorithms entice exploration logically.”

    From Playtesting: What is Beyond Personas (2023) · cites this paper as [13]

  • APF may also be formulated on other exploration methods such as exemplar models.

    “Last but not least, though we presented the APF on top of exploration methods CTS and ICM, APF may also be formulated on other exploration methods such as exemplar models [13].”

    From Playtesting: What is Beyond Personas (2023) · cites this paper as [13]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 30

  1. Abel, David, Agarwal, Alekh, Diaz, Fernando, Krishnamurthy, Akshay, and Schapire, Robert E. Exploratory gradient boosting for reinforcement learning in complex domains. In Advances in Neural Information Processing Systems (NIPS) , 2016.
  2. Achiam, Joshua and Sastry, Shankar. Surprise-based intrinsic motivation for deep reinforcement learning. CoRR , abs/1703.01732, 2017. · arXiv
  3. Barto, Andrew G. and Mahadevan, Sridhar. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems , 13(1-2), 2003.
  4. Bellemare, Marc G., Srinivasan, Sriram, Ostrovski, Georg, Schaul, Tom, Saxton, David, and Munos, Remi. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems (NIPS) , 2016.
  5. Brafman, Ronen I. and Tennenholtz, Moshe. R-max - a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research (JMLR) , 2002.
  6. Chapelle, O. and Li, Lihong. An empirical evaluation of thompson sampling. In Advances in Neural Information Processing Systems (NIPS) , 2011.
  7. Chentanez, Nuttapong, Barto, Andrew G, and Singh, Satinder P. Intrinsically Motivated Reinforcement Learning. In Advances in Neural Information Processing Systems (NIPS) . MIT Press, 2005.
  8. Duan, Yan, Chen, Xi, Houthooft, Rein, Schulman, John, and Abbeel, Pieter. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning (ICML) , 2016.
  9. Florensa, Carlos Campo, Duan, Yan, and Abbeel, Pieter. Stochastic neural networks for hierarchical reinforcement learning. In International Conference on Learning Representations (ICLR) , 2017.
  10. Goodfellow, Ian, Pouget-Abadie, Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair, Sherjil, Courville, Aaron, and Bengio, Yoshua. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS) . 2014.
  11. Heess, Nicolas, Wayne, Gregory, Tassa, Yuval, Lillicrap, Timothy P., Riedmiller, Martin A., and Silver, David. Learning and transfer of modulated locomotor controllers. CoRR , abs/1610.05182, 2016. · arXiv
  12. Houthooft, Rein, Chen, Xi, Duan, Yan, Schulman, John, Turck, Filip De, and Abbeel, Pieter. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems (NIPS) , 2016.
  13. Kakade, Sham, Kearns, Michael, and Langford, John. Exploration in metric state spaces. In International Conference on Machine Learning (ICML) , 2003.
  14. Kolter, J. Zico and Ng, Andrew Y. Near-bayesian exploration in polynomial time. In International Conference on Machine Learning (ICML) , 2009.
  15. Kulkarni, Tejas D, Narasimhan, Karthik, Saeedi, Ardavan, and Tenenbaum, Josh. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in Neural Information Processing Systems (NIPS) . 2016.
  16. Lillicrap, Timothy P., Hunt, Jonathan J., Pritzel, Alexander, Heess, Nicolas, Erez, Tom, Tassa, Yuval, Silver, David, and Wierstra, Daan. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR) , 2015.
  17. Malisiewicz, Tomasz, Gupta, Abhinav, and Efros, Alexei A. Ensemble of exemplar-svms for object detection and beyond. In International Conference on Computer Vision (ICCV) , 2011.
  18. Mathieu, Michaël, Couprie, Camille, and LeCun, Yann. Deep multi-scale video prediction beyond mean square error. CoRR , abs/1511.05440, 2015. URL http://arxiv.org/abs/1511.05440 . · arXiv
  19. Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A., Veness, Joel, Bellemare, Marc G., Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K., Ostrovski, Georg, Petersen, Stig, Beattie, Charles, Sadik, Amir, Antonoglou, Ioannis, King, Helen, Kumaran, Dharshan, Wierstra, Daan, Legg, Shane, and Hassabis, Demis. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 02 2015.
  20. Oh, Junhyuk, Guo, Xiaoxiao, Lee, Honglak, Lewis, Richard, and Singh, Satinder. Action-conditional video prediction using deep networks in atari games. In Advances in Neural Information Processing Systems (NIPS) , 2015.
  21. Osband, Ian, Blundell, Charles, and Alexander Pritzel, Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Advances in Neural Information Processing Systems (NIPS) , 2016.
  22. Pathak, Deepak, Agrawal, Pulkit, Efros, Alexei A., and Darrell, Trevor. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning (ICML) , 2017.
  23. Pazis, Jason and Parr, Ronald. Pac optimal exploration in continuous space markov decision processes. In AAAI Conference on Artificial Intelligence (AAAI) , 2013.
  24. Salimans, Tim, Goodfellow, Ian J., Zaremba, Wojciech, Cheung, Vicki, Radford, Alec, and Chen, Xi. Improved techniques for training gans. In Advances in Neural Information Processing Systems (NIPS) , 2016.
  25. Schmidhuber, Jürgen. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proceedings of the First International Conference on Simulation of Adaptive Behavior on From Animals to Animats , Cambridge, MA, USA, 1990. MIT Press. ISBN 0-26263138-5.
  26. Schulman, John, Levine, Sergey, Moritz, Philipp, Jordan, Michael I., and Abbeel, Pieter. Trust region policy optimization. In International Conference on Machine Learning (ICML) , 2015.
  27. Stadie, Bradly C., Levine, Sergey, and Abbeel, Pieter. Incentivizing exploration in reinforcement learning with deep predictive models. CoRR , abs/1507.00814, 2015. · arXiv
  28. Stolle, Martin and Precup, Doina. Learning Options in Reinforcement Learning . Springer Berlin Heidelberg, Berlin, Heidelberg, 2002. ISBN 978-3-540-45622-3. doi: 10.1007/3-540-45622-8\_16. · DOI
  29. Strehl, Alexander L. and Littman, Michael L. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences , 2009.
  30. Tang, Haoran, Houthooft, Rein, Foote, Davis, Stooke, Adam, Chen, Xi, Duan, Yan, Schulman, John, Turck, Filip De, and Abbeel, Pieter. #exploration: A study of count-based exploration for deep reinforcement learning. CoRR , abs/1611.04717, 2016. · arXiv

Cited by

Sorted by citations

Authors
Fu, J., Co-Reyes, J. D., & Levine, S.
Affiliations
University of California, Berkeley (US)
ORCID
Levine, Sergey
Venue
arXiv
Publisher
arXiv
Published
2017
Volume
30
Pages
2577-2587
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1703.01260
arXiv
1703.01260
OpenAlex
W2604373826
Cited by
95
References
0
Citations by year
2025: 1 · 2024: 6 · 2023: 11 · 2022: 11 · 2021: 12 · 2020: 20 · 2019: 17 · 2018: 14 · 2017: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1614653, 1700696)
  • Office of Naval Research
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
Keywords
exploration, deep reinforcement learning, novelty detection, exemplar models, discriminative classifiers, sparse rewards, high-dimensional observations, egocentric vision, ViZDoom

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review