CoChat
Sign Up
PreprintCited by 24Reinforcement Learning in RoboticsAdvanced Bandit Algorithms ResearchArtificial Intelligence in Games

Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learning

Open access
Janz, David, Hron, Jiri, Mazur, Przemysław, Hofmann, Katja, Hernández-Lobato, José Miguel, Tschiatschek, Sebastian·2018·arXiv
DOI 10.48550/arxiv.1810.06530arXiv 1810.06530
Open this paper in CoChat — free accountOpen PDF

Abstract

Posterior sampling for reinforcement learning (PSRL) is an effective method for balancing exploration and exploitation in reinforcement learning. Randomised value functions (RVF) can be viewed as a promising approach to scaling PSRL. However, we show that most contemporary algorithms combining RVF with neural network function approximation do not possess the properties which make PSRL effective, and provably fail in sparse reward problems. Moreover, we find that propagation of uncertainty, a property of PSRL previously thought important for exploration, does not preclude this failure. We use these insights to design Successor Uncertainties (SU), a cheap and easy to implement RVF algorithm that retains key properties of PSRL. SU is highly effective on hard tabular exploration benchmarks. Furthermore, on the Atari 2600 domain, it surpasses human performance on 38 of 49 games tested (achieving a median human normalised score of 2.09), and outperforms its closest RVF competitor, Bootstrapped DQN, on 36 of those.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1810.06530

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • DNNs are not robust to adversarial examples.

    “Despite the success of DNNs in dealing with complex and high-dimensional image data, they are not robust to adversarial examples [156].”

    From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [156]

    Section 3 (p. 4) discusses difficulties with neural-network function approximation in reinforcement learning, while Appendix C.2 (p. 19) describes a network that processes Atari images. Neither addresses robustness to adversarial examples; searches for adversarial examples, attacks, and perturbations found no supporting passage in our copy. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 32

  1. Auer, P. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422, 2002.
  2. Azizzadenesheli, K., Brunskill, E., and Anandkumar, A. Efficient exploration through bayesian deep Q-networks. arXiv preprint arXiv:1802.04412 , 2018. · arXiv
  3. Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., van Hasselt, H. P., and Silver, D. Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS) , 2017.
  4. Busoniu, L., Babuska, R., Schutter, B. D., and Ernst, D. Reinforcement Learning and Dynamic Programming Using Function Approximators . CRC Press, 2010.
  5. Dayan, P. Improving generalization for temporal difference learning: the successor representation. Neural Computation , 5(4):613-624, 1993.
  6. Dearden, R., Friedman, N., and Russell, S. J. Bayesian Q-Learning. In AAAI/IAAI , pp. 761-768. AAAI Press / The MIT Press, 1998.
  7. Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., and Clune, J. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995 , 2019. · arXiv
  8. Gal, Y. Uncertainty in deep learning . PhD thesis, University of Cambridge, 2016.
  9. Hessel, M., Modayil, J., van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M. G., and Silver, D. Rainbow: combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence , 2018.
  10. Kaelbling, L. P., Littman, M. L., and Moore, A. W. Reinforcement learning: a survey. Journal of artificial intelligence research , 4:237-285, 1996.
  11. Kingma, D. P. and Ba, J. Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
  12. Kulkarni, T. D., Saeedi, A., Gautam, S., and Gershman, S. J. Deep successor reinforcement learning. arXiv preprint arXiv:1606.02396 , 2016. · arXiv
  13. Levine, N., Zahavy, T., Mankowitz, D. J., Tamar, A., and Mannor, S. Shallow updates for deep reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS) , 2017.
  14. Lipton, Z. C., Li, X., Gao, J., Li, L., Ahmed, F., and Deng, L. BBQ-Networks: efficient exploration in deep reinforcement learning for task-oriented dialogue systems. In AAAI Conference on Artificial Intelligence , 2018.
  15. Machado, M. C., Rosenbaum, C., Guo, X., Liu, M., Tesauro, G., and Campbell, M. Eigenoption discovery through the deep successor representation. arXiv preprint arXiv:1710.11089 , 2017. · arXiv
  16. Machado, M. C., Bellemare, M. G., and Bowling, M. Count-based exploration with the successor representation. arXiv preprint arXiv:1807.11622 , 2018. · arXiv
  17. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature , 518(7540):529, 2015.
  18. Moerland, T. M., Broekens, J., and Jonker, C. M. Efficient exploration with double uncertain value networks. arXiv preprint arXiv:1711.10789 , 2017. · arXiv
  19. Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS) , 2016.
  20. O'Donoghue, B., Osband, I., Munos, R., and Mnih, V. The uncertainty Bellman equation and exploration. In International Conference on Machine Learning (ICML) , 2018.
  21. Osband, I. and Van Roy, B. On lower bounds for regret in reinforcement learning. arXiv preprint arXiv:1608.02732 , 2016. · arXiv
  22. Osband, I., Russo, D., and Van Roy, B. (More) efficient reinforcement learning via posterior sampling. In Advances in Neural Information Processing Systems , 2013.
  23. Osband, I., Blundell, C., Pritzel, A., and Van Roy, B. Deep exploration via bootstrapped DQN. In Advances in Neural Information Processing Systems (NeurIPS) , 2016a.
  24. Osband, I., Van Roy, B., and Wen, Z. Generalization and exploration via randomized value functions. International Conference on Machine Learning (ICML) , 2016b.
  25. Osband, I., Aslanides, J., and Cassirer, A. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS) , 2018.
  26. Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R. Y., Chen, X., Asfour, T., Abbeel, P., and Andrychowicz, M. Parameter space noise for exploration. In International Conference on Learning Representations (ICLR) , 2018.
  27. Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. arXiv preprint arXiv:1511.05952 , 2015. · arXiv
  28. Strens, M. A Bayesian framework for reinforcement learning. In Conference on Machine Learning (ICML) , 2000.
  29. Sutton, R. S., Barto, A. G., et al. Reinforcement learning: An introduction . MIT press, 1998.
  30. Touati, A., Satija, H., Romoff, J., Pineau, J., and Vincent, P. Randomized value functions via multiplicative normalizing flows. arXiv preprint arXiv:1806.02315 , 2018. · arXiv
  31. Van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double Q-learning. In AAAI Conference on Artificial Intelligence , 2016.
  32. Wang, Z., Schaul, T., Hessel, M., Van Hasselt, H., Lanctot, M., and De Freitas, N. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learning (ICML) , 2016.

Cited by

Sorted by citations

Related

Authors
Janz, D., Hron, J., Mazur, P., Hofmann, K., Hernández-Lobato, J. M., & Tschiatschek, S.
Affiliations
  • University of Cambridge (GB)
  • Microsoft (United States) (US)
  • Turing Institute (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
Volume
32
Pages
4507-4516
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1810.06530
arXiv
1810.06530
OpenAlex
W2971265002
Cited by
23
References
0
Citations by year
2025: 1 · 2024: 1 · 2023: 4 · 2022: 2 · 2021: 7 · 2020: 7 · 2019: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Cambridge University Engineering Department Publications Database (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
Keywords
temporal difference learning, exploration-exploitation balance, error propagation, neural network function approximation, Atari 2600

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review