CoChat
Sign Up
Cited by 53Reinforcement Learning in RoboticsArtificial Intelligence in GamesDomain Adaptation and Few-Shot Learning

An Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey

Open access
Aubret, Arthur, Matignon, Laetitia, Hassas, Salima·2023·Entropy
DOI 10.3390/e25020327PMC PMC9954873
Open this paper in CoChat — free accountOpen PDF

Abstract

The reinforcement learning (RL) research area is very active, with an important number of new contributions, especially considering the emergent field of deep RL (DRL). However, a number of scientific and technical challenges still need to be resolved, among which we acknowledge the ability to abstract actions or the difficulty to explore the environment in sparse-reward settings which can be addressed by intrinsic motivation (IM). We propose to survey these research works through a new taxonomy based on information theory: we computationally revisit the notions of surprise, novelty, and skill-learning. This allows us to identify advantages and disadvantages of methods and exhibit current outlooks of research. Our analysis suggests that novelty and surprise can assist the building of a hierarchy of transferable skills which abstracts dynamics and makes the exploration process more robust.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/e25020327

Cites

138 of 139 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Many models in modern robotics and machine learning treat intrinsic motivation primarily as an exploratory mechanism for improving reward acquisition efficiency in environments with sparse external rewards.

    “many models proposed in modern robotics and machine learning tend to treat intrinsic motivation primarily as an exploratory mechanism for improving reward acquisition efficiency in environments with sparse external rewards (Aubret et al. 2023).”

    From Emergence of Goal‐Directed Behaviors via Active Inference With Self‐Prior (2026) · cites this paper as Aubret et al. 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 140 references couldn’t be matched to a known paper.

Research works printed but not linked: 51

  1. [2]Bellemare M.G. Naddaf Y. Veness J. Bowling M. The Arcade Learning Environment: An Evaluation Platform for General Agents (Extended Abstract) Proceedings of the IJCAI AAAI Press Washington, DC, USA 2015 4148 4152
  2. [11]Baldassarre G. Mirolli M. Intrinsically motivated learning systems: An overview Intrinsically Motivated Learning in Natural and Artificial Systems Springer Berlin/Heidelberg, Germany 2013 1 14
  3. [12]Colas C. Karch T. Sigaud O. Oudeyer P.Y. Intrinsically Motivated Goal-Conditioned Reinforcement Learning: A Short Survey arXiv 2020 2012.09830
  4. [13]Amin S. Gomrokchi M. Satija H. van Hoof H. Precup D. A Survey of Exploration Methods in Reinforcement Learning arXiv 2021 2109.00157
  5. [14]Baldassarre G. Intrinsic motivations and open-ended learning arXiv 2019 1912.13263
  6. [17]Schmidhuber J. Driven by compression progress: A simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes Proceedings of the Workshop on Anticipatory Behavior in Adaptive Learning Systems Springer Berlin/Heidelberg, Germany 2008 48 76
  7. [18]Salge C. Glackin C. Polani D. Empowerment—An introduction Guided Self-Organization: Inception Springer Berlin/Heidelberg, Germany 2014 67 114
  8. [22]Cuervo S. Alzate M. Emergent cooperation through mutual information maximization arXiv 2020 2006.11769
  9. [23]Sperati V. Trianni V. Nolfi S. Mutual information as a task-independent utility function for evolutionary robotics Guided Self-Organization: Inception Springer Berlin/Heidelberg, Germany 2014 389 414
  10. [26]Puterman M.L. Markov Decision Processes: Discrete Stochastic Dynamic Programming John Wiley & Sons Hoboken, NJ, USA 2014
  11. [32]Oudeyer P.Y. Kaplan F. How can we define intrinsic motivation? Proceedings of the 8th International Conference on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems Lund University Cognitive Studies LUCS, Brighton Lund, Sweden 2008
  12. [36]Cover T.M. Thomas J.A. Elements of Information Theory John Wiley & Sons Hoboken, NJ, USA 2012
  13. [43]Lillicrap T.P. Hunt J.J. Pritzel A. Heess N. Erez T. Tassa Y. Silver D. Wierstra D. Continuous control with deep reinforcement learning arXiv 2015 1509.02971
  14. [45]Plappert M. Houthooft R. Dhariwal P. Sidor S. Chen R.Y. Chen X. Asfour T. Abbeel P. Andrychowicz M. Parameter space noise for exploration arXiv 2017 1706.01905
  15. [47]Fortunato M. Azar M.G. Piot B. Menick J. Osband I. Graves A. Mnih V. Munos R. Hassabis D. Pietquin O. Noisy networks for exploration arXiv 2017 1706.10295
  16. [51]Amodei D. Olah C. Steinhardt J. Christiano P. Schulman J. Mané D. Concrete problems in AI safety arXiv 2016 1606.06565
  17. [55]Heess N. Wayne G. Tassa Y. Lillicrap T. Riedmiller M. Silver D. Learning and transfer of modulated locomotor controllers arXiv 2016 1610.05182
  18. [57]Nachum O. Tang H. Lu X. Gu S. Lee H. Levine S. Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning? arXiv 2019 1909.10618
  19. [59]Matusch B. Ba J. Hafner D. Evaluating agents without rewards arXiv 2020 2012.11538
  20. [60]Ekman P.E. Davidson R.J. The Nature of Emotion: Fundamental Questions Oxford University Press Oxford, UK 1994
  21. [63]Stadie B.C. Levine S. Abbeel P. Incentivizing exploration in reinforcement learning with deep predictive models arXiv 2015 1507.00814
  22. [69]Efroni Y. Misra D. Krishnamurthy A. Agarwal A. Langford J. Provably Filtering Exogenous Distractors using Multistep Inverse Dynamics Proceedings of the International Conference on Learning Representations Virtual 18–24 July 2021
  23. [71]Azar M.G. Piot B. Pires B.A. Grill J.B. Altché F. Munos R. World discovery models arXiv 2019 1902.07685
  24. [79]Houthooft R. Chen X. Duan Y. Schulman J. De Turck F. Abbeel P. Vime: Variational information maximizing exploration Proceedings of the Advances in Neural Information Processing Systems Barcelona, Spain 5–10 December 2016 1109 1117
  25. [81]Blundell C. Cornebise J. Kavukcuoglu K. Wierstra D. Weight uncertainty in neural networks arXiv 2015 1505.05424
  26. [82]Achiam J. Sastry S. Surprise-based intrinsic motivation for deep reinforcement learning arXiv 2017 1703.01732
  27. [91]Becker-Ehmck P. Karl M. Peters J. van der Smagt P. Exploration via Empowerment Gain: Combining Novelty, Surprise and Learning Progress Proceedings of the ICML 2021 Workshop on Unsupervised Reinforcement Learning Virtual Event 18–24 July 2021
  28. [92]Lehman J. Stanley K.O. Novelty search and the problem with objectives Genetic Programming Theory and Practice IX Springer Berlin/Heidelberg, Germany 2011 37 56
  29. [107]Zhang T. Xu H. Wang X. Wu Y. Keutzer K. Gonzalez J.E. Tian Y. BeBold: Exploration Beyond the Boundary of Explored Regions arXiv 2020 2012.08621
  30. [109]Islam R. Seraj R. Bacon P.L. Precup D. Entropy regularization with discounted future state distribution in policy gradient methods arXiv 2019 1912.05104
  31. [110]Lee L. Eysenbach B. Parisotto E. Xing E. Levine S. Salakhutdinov R. Efficient Exploration via State Marginal Matching arXiv 2019 1906.05274
  32. [112]Vezzani G. Gupta A. Natale L. Abbeel P. Learning latent state representation for speeding up exploration arXiv 2019 1905.12621
  33. [114]Aubret A. Matignon L. Hassas S. DisTop: Discovering a Topological representation to learn diverse and rewarding skills arXiv 2021 2106.03853
  34. [115]Guo Z.D. Azar M.G. Saade A. Thakoor S. Piot B. Pires B.A. Valko M. Mesnard T. Lattimore T. Munos R. Geometric entropic exploration arXiv 2021 2101.02055
  35. [120]Liu H. Abbeel P. Behavior from the void: Unsupervised active pre-training arXiv 2021 2103.04551
  36. [121]Srinivas A. Laskin M. Abbeel P. Curl: Contrastive unsupervised representations for reinforcement learning arXiv 2020 2004.04136
  37. [125]Yarats D. Fergus R. Lazaric A. Pinto L. Reinforcement learning with prototypical representations arXiv 2021 2102.11271
  38. [129]Guo Z.D. Thakoor S. Pîslar M. Pires B.A. Altché F. Tallec C. Saade A. Calandriello D. Grill J.B. Tang Y. Byol-explore: Exploration by bootstrapped prediction arXiv 2022 2206.08332
  39. [134]Stanton C. Clune J. Deep Curiosity Search: Intra-Life Exploration Can Improve Performance on Challenging Deep Reinforcement Learning Problems arXiv 2018 1806.00553
  40. [139]Alemi A.A. Fischer I. Dillon J.V. Murphy K. Deep variational information bottleneck arXiv 2016 1612.00410
  41. [140]Eysenbach B. Gupta A. Ibarz J. Levine S. Diversity is All You Need: Learning Skills without a Reward Function arXiv 2018 1802.06070
  42. [144]Achiam J. Edwards H. Amodei D. Abbeel P. Variational option discovery algorithms arXiv 2018 1807.10299
  43. [147]Gregor K. Rezende D.J. Wierstra D. Variational intrinsic control arXiv 2016 1611.07507
  44. [167]Racaniere S. Lampinen A.K. Santoro A. Reichert D.P. Firoiu V. Lillicrap T.P. Automated curricula through setter-solver interactions arXiv 2019 1909.12892
  45. [168]Goodfellow I. Pouget-Abadie J. Mirza M. Xu B. Warde-Farley D. Ozair S. Courville A. Bengio Y. Generative adversarial nets Proceedings of the Advances in Neural Information Processing Systems Montreal, QC, Canada 8–13 December 2014 2672 2680
  46. [171]Zhao R. Tresp V. Curiosity-driven experience prioritization via density estimation arXiv 2019 1902.08039
  47. [173]Kovač G. Laversanne-Finot A. Oudeyer P.Y. Grimgep: Learning progress for robust goal sampling in visual deep reinforcement learning arXiv 2020 2008.04388
  48. [175]Li S. Zhang J. Wang J. Zhang C. Efficient Hierarchical Exploration with Stable Subgoal Representation Learning arXiv 2021 2105.14750
  49. [177]Ecoffet A. Huizinga J. Lehman J. Stanley K.O. Clune J. Go-Explore: A New Approach for Hard-Exploration Problems arXiv 2019 1901.10995
  50. [184]Konczak J. Neural Development and Sensorimotor Control 2004 Available online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3075656 (accessed on 1 February 2023)
  51. [186]Oudeyer P.Y. Baranes A. Kaplan F. Intrinsically motivated learning of real-world sensorimotor skills with developmental constraints Intrinsically Motivated Learning in Natural and Artificial Systems Springer Berlin/Heidelberg, Germany 2013 303 365

Cited by

Sorted by citations

Related

Authors
Aubret, A., Matignon, L., & Hassas, S.
Affiliations
  • Lyon 1 Université (FR)
  • Univ Lyon, UCBL, CNRS, INSA Lyon, LIRIS, UMR5205, 69622 Villeurbanne, France
Corresponding author
Aubret, Arthur
ORCID
Aubret, Arthur
Venue
Entropy
Publisher
MDPI AG
Published
2023-02-10
Volume
25
Issue
2
Pages
327
ISSN
1099-4300 (ISSN-L)
DOI
10.3390/e25020327
PMC
PMC9954873
OpenAlex
W4319966548
Cited by
52
References
140
Citations by year
2026: 12 · 2025: 15 · 2024: 14 · 2023: 9 · 2022: 1 · 2019: 1
FWCI
1.05
Type
journal article
Language
en
Open access
Licence
Locations
  • Entropy (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • HAL (submittedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
ANR project DeLiCio (ANR-19-CE23-0006)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
intrinsic motivation, reinforcement learning, information theory, surprise, novelty, skill acquisition, sparse rewards, action abstraction, exploration, deep reinforcement learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review