CoChat
Sign Up
PreprintCited by 401Reinforcement Learning in RoboticsArtificial Intelligence in GamesPsychiatry, Mental Health, Neuroscience

Large-Scale Study of Curiosity-Driven Learning

Open access
Burda, Yuri, Edwards, Harri, Pathak, Deepak, Storkey, Amos, Darrell, Trevor, Efros, Alexei A.·2018·arXiv
DOI 10.48550/arxiv.1808.04355arXiv 1808.04355
Open this paper in CoChat — free accountOpen PDF

Abstract

Reinforcement learning algorithms rely on carefully engineering environment rewards that are extrinsic to the agent. However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent. Curiosity is a type of intrinsic reward function which uses prediction error as reward signal. In this paper: (a) We perform the first large-scale study of purely curiosity-driven learning, i.e. without any extrinsic rewards, across 54 standard benchmark environments, including the Atari game suite. Our results show surprisingly good performance, and a high degree of alignment between the intrinsic curiosity objective and the hand-designed extrinsic rewards of many game environments. (b) We investigate the effect of using different feature spaces for computing prediction error and show that random features are sufficient for many popular RL game benchmarks, but learned features appear to generalize better (e.g. to novel game levels in Super Mario Bros.). (c) We demonstrate limitations of the prediction-based rewards in stochastic setups. Game-play videos and code are at https://pathak22.github.io/large-scale-curiosity/

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1808.04355

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 36

  1. [2][2] J. Achiam and S. Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv:1703.01732 , 2017. - · arXiv
  2. [3][3] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In NIPS , 2016. -
  3. [4][4] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279, jun 2013. -
  4. [5][5] R. Y. Chen, J. Schulman, P. Abbeel, and S. Sidor. UCB and infogain exploration via q -ensembles. arXiv:1706.01502 , 2017. - · arXiv
  5. [6][6] G. Costikyan. Uncertainty in games . Mit Press, 2013. -
  6. [7][7] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint , 2018. -
  7. [8][8] M. Fortunato, M. G. Azar, B. Piot, J. Menick, I. Osband, A. Graves, V. Mnih, R. Munos, D. Hassabis, O. Pietquin, C. Blundell, and S. Legg. Noisy networks for exploration. arXiv:1706.10295 , 2017. - · arXiv
  8. [9][9] J. Fu, J. D. Co-Reyes, and S. Levine. EX2: Exploration with exemplar models for deep reinforcement learning. NIPS , 2017. -
  9. [10][10] K. Gregor, D. J. Rezende, and D. Wierstra. Variational intrinsic control. ICLR Workshop , 2017. -
  10. [11][11] R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel. Vime: Variational information maximizing exploration. In NIPS , 2016. -
  11. [12][12] R. Hunicke, M. LeBlanc, and R. Zubek. Mda: A formal approach to game design and game research. In AAAI Workshop on Challenges in Game AI , 2004. -
  12. [13][13] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 , 2015. - · arXiv
  13. [15][15] D. P. Kingma and M. Welling. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 , 2013. - · arXiv
  14. [16][16] N. Lazzaro. Why we play games: Four keys to more emotion in player experiences. In Proceedings of GDC , 2004. -
  15. [19][19] M. Lopes, T. Lang, M. Toussaint, and P.-Y. Oudeyer. Exploration in model-based reinforcement learning by empirically estimating learning progress. In NIPS , 2012. -
  16. [20][20] M. Lopes, T. Lang, M. Toussaint, and P.-Y. Oudeyer. Exploration in model-based reinforcement learning by empirically estimating learning progress. In NIPS , 2012. -
  17. [22][22] S. Mohamed and D. J. Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In NIPS , 2015. -
  18. [23][23] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped dqn. In NIPS , 2016. -
  19. [24][24] G. Ostrovski, M. G. Bellemare, A. v. d. Oord, and R. Munos. Count-based exploration with neural density models. arXiv:1703.01310 , 2017. - · arXiv
  20. [25][25] P.-Y. Oudeyer. Computational theories of curiosity-driven learning. arXiv preprint arXiv:1802.10546 , 2018. - · arXiv
  21. [27][27] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self-supervised prediction. In ICML , 2017. -
  22. [28][28] D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell. Zero-shot visual imitation. In ICLR , 2018. -
  23. [29][29] M. Plappert, R. Houthooft, P. Dhariwal, S. Sidor, R. Y. Chen, X. Chen, T. Asfour, P. Abbeel, and M. Andrychowicz. Parameter space noise for exploration. arXiv:1706.01905 , 2017. - · arXiv
  24. [30][30] P. Poupart, N. Vlassis, J. Hoey, and K. Regan. An analytic solution to discrete bayesian reinforcement learning. In ICML , 2006. -
  25. [31][31] D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082 , 2014. - · arXiv
  26. [33][33] A. M. Saxe, P. W. Koh, Z. Chen, M. Bhand, B. Suresh, and A. Y. Ng. On random weights and unsupervised feature learning. In ICML , pages 1089-1096, 2011. -
  27. [34][34] J. Schmidhuber. Curious model-building control systems. In Neural Networks, 1991. 1991 IEEE International Joint Conference on , pages 1458-1463. IEEE, 1991. -
  28. [35][35] J. Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In From animals to animats: Proceedings of the first international conference on simulation of adaptive behavior , 1991. -
  29. [37][37] J. Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions on Autonomous Mental Development , 2010. -
  30. [39][39] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. - · arXiv
  31. [40][40] S. P. Singh, A. G. Barto, and N. Chentanez. Intrinsically motivated reinforcement learning. In NIPS , 2005. -
  32. [42][42] B. C. Stadie, S. Levine, and P. Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. NIPS Workshop , 2015. -
  33. [43][43] K. O. Stanley and J. Lehman. Why greatness cannot be planned: The myth of the objective . Springer, 2015. -
  34. [45][45] S. Sukhbaatar, I. Kostrikov, A. Szlam, and R. Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. In ICLR , 2018. -
  35. [46][46] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction . MIT press Cambridge, 1998. -
  36. [47][47] H. Tang, R. Houthooft, D. Foote, A. Stooke, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel. #Exploration: A study of count-based exploration for deep reinforcement learning. Advances in Neural Information Processing Systems , 2017. -

Cited by

Sorted by citations

Authors
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., & Efros, A. A.
Affiliations
  • OpenAI (United States) (US)
  • University of California, Berkeley (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1808.04355
arXiv
1808.04355
OpenAlex
W2885550588
Cited by
401
References
6
Citations by year
2026: 1 · 2025: 15 · 2024: 49 · 2023: 54 · 2022: 62 · 2021: 74 · 2020: 83 · 2019: 57 · 2018: 6
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
Keywords
curiosity-driven learning, intrinsic rewards, prediction error, curiosity-driven exploration, Atari games, reward function design, feature space, learned features, random features, Super Mario Bros., stochastic environments, benchmark environments

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review