The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Intrinsically Motivated Reinforcement Learning: An Evolutionary Perspective
Abstract
There is great interest in building intrinsic motivation into artificial systems using the reinforcement learning framework. Yet, what intrinsic motivation may mean computationally, and how it may differ from extrinsic motivation, remains a murky and controversial subject. In this paper, we adopt an evolutionary perspective and define a new optimal reward framework that captures the pressure to design good primary reward functions that lead to evolutionary success across environments. The results of two computational experiments show that optimal primary reward signals may yield both emergent intrinsic and extrinsic motivation. The evolutionary perspective and the associated optimal reward framework thus lead to the conclusion that there are no hard and fast features distinguishing intrinsic and extrinsic reward computationally. Rather, the directness of the relationship between rewarding behavior and evolutionary success varies along a continuum.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Self-citationIntrinsic Motivation and Self-Determination in Human Behavior
Deci, E. L., & Ryan, R. M. (1985) · doi:10.1007/978-1-4899-2271-7 · Cited by 26,772
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions
Ryan, R. M., & Deci, E. L. (2000) · Contemporary Educational Psychology · doi:10.1006/ceps.1999.1020 · Cited by 18,666
Intrinsic Motivation and Self-Determination in Human Behavior
Miller, K. A., Deci, E. L., & Ryan, R. M. (1988) · Contemporary Sociology · doi:10.2307/2070638 · Cited by 15,312
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Skinner, B. F., & Hull, C. L. (1944) · The American Journal of Psychology · doi:10.2307/1416955 · Cited by 4,051
Cited by
Neuronal Reward and Decision Signals: From Theories to Data
Schultz, W. (2015) · Physiological Reviews · doi:10.1152/physrev.00023.2014 · Cited by 1,126
Information-seeking, curiosity, and attention: computational and neural mechanisms
Gottlieb, J., Oudeyer, P.-Y., Lopes, M., & Baranes, A. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.09.001 · Cited by 784
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Kulkarni, T. D., Narasimhan, K., Saeedi, A., & Tenenbaum, J. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06057 · Cited by 770
Human-level performance in 3D multiplayer games with population-based reinforcement learning
Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L., Lever, G., Castañeda, A. G., Beattie, C., Rabinowitz, N. C., Morcos, A. S., Ruderman, A., et al. (2019) · Science · doi:10.1126/science.aau6249 · Cited by 662
Tamar, A., Wu, Y., Thomas, G., Levine, S., & Abbeel, P. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/700 · Cited by 484
Active learning of inverse models with intrinsically motivated goal exploration in robots
Baranes, A., & Oudeyer, P.-Y. (2013) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2012.05.008 · Cited by 455
Silver, D., Singh, S., Precup, D., & Sutton, R. S. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103535 · Cited by 445
Self-citationEmergent Tool Use From Multi-Agent Autocurricula
Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019) · arXiv · doi:10.48550/arxiv.1909.07528 · Cited by 424
“Intrinsic motivation methods have been widely studied in the literature [35, 36].”
BackgroundMethodMechanisms of motivation–cognition interaction: challenges and opportunities
Braver, T. S., Krug, M. K., Chiew, K. S., Kool, W., Westbrook, J. A., Clement, N. J., Adcock, R. A., Barch, D. M., Botvinick, M. M., Carver, C. S., et al. (2014) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-014-0300-0 · Cited by 391
Intrinsic motivation, curiosity, and learning
Oudeyer, P.-Y., Gottlieb, J., & Lopes, M. (2016) · Progress in Brain Research · doi:10.1016/bs.pbr.2016.05.005 · Cited by 387
Reinforcement learning improves behaviour from evaluative feedback
Littman, M. L. (2015) · Nature · doi:10.1038/nature14540 · Cited by 368
Deep reinforcement learning and its neuroscientific implications
Botvinick, M., Wang, J. X., Dabney, W., Miller, K. J., & Kurth-Nelson, Z. (2020) · Neuron · doi:10.1016/j.neuron.2020.06.014 · Cited by 269
Dopamine neurons encode performance error in singing birds
Gadagkar, V., Puzerey, P. A., Chen, R., Baird-Daniel, E., Farhang, A. R., & Goldberg, J. H. (2016) · Science · doi:10.1126/science.aah6837 · Cited by 268
Computational Models of Interoception and Body Regulation
Petzschner, F. H., Garfinkel, S. N., Paulus, M. P., Koch, C., & Khalsa, S. S. (2021) · Trends in Neurosciences · doi:10.1016/j.tins.2020.09.012 · Cited by 227
Homeostatic reinforcement learning for integrating reward collection and physiological stability
Keramati, M., & Gutkin, B. (2014) · eLife · doi:10.7554/elife.04811 · Cited by 218
Emotion in reinforcement learning agents and robots: a survey
Moerland, T. M., Broekens, J., & Jonker, C. M. (2018) · Machine Learning · doi:10.1007/s10994-017-5666-0 · Cited by 205
Brewer, J. A., Ruf, A., Beccia, A. L., Essien, G. I., Finn, L. M., Lutterveld, R. V., & Mason, A. E. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01418 · Cited by 82
Intrinsic motivations and open-ended development in animals, humans, and robots: an overview
Baldassarre, G., Stafford, T., Mirolli, M., Redgrave, P., Ryan, R. M., & Barto, A. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00985 · Cited by 73
Self-citationAn Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey
Aubret, A., Matignon, L., & Hassas, S. (2023) · Entropy · doi:10.3390/e25020327 · Cited by 52
Scalar reward is not enough: a response to Silver, Singh, Precup and Sutton (2021)
Vamplew, P., Smith, B. J., Källström, J., Ramos, G., Rădulescu, R., Roijers, D. M., Hayes, C. F., Heintz, F., Mannion, P., Libin, P. J. K., et al. (2022) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-022-09575-5 · Cited by 41
“Conceivably this could also arise in the context of computational intelligence, where agents based on evolutionary algorithms or reinforcement learning might construct their own internal reward signals to guide their learning and decision-making [32, 71, 76].”
BackgroundGeneral intelligence requires rethinking exploration
Jiang, M., Rocktäschel, T., & Grefenstette, E. (2022) · arXiv · doi:10.48550/arxiv.2211.07819 · Cited by 17
Exploration and preference satisfaction trade-off in reward-free learning
Sajid, N., Tigas, P., Zakharov, A., Fountas, Z., & Friston, K. (2021) · arXiv · doi:10.48550/arxiv.2106.04316 · Cited by 11
“In reality, agents do not operate in a problem-solving setting, where a “critic” may not be readily available to provide immediate rewards [5, 59, 60].”
BackgroundImproving Environment Novelty Quantification for Effective Unsupervised Environment Design
Teoh, J., Li, W., & Varakantham, P. (2025) · arXiv
“Curiosity-driven Approaches in RL CENIE and curiosity-driven RL [47, 50] share a conceptual similarity in leveraging novelty or unfamiliarity to guide learning.”
Improving Environment Novelty Quantification for Effective Unsupervised Environment Design
Teoh, J., Li, W., & Varakantham, P. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-4300
Metadata
- Authors
- Singh, S., Lewis, R. L., Barto, A. G., & Sorg, J.
- Venue
- IEEE Transactions on Autonomous Mental Development
- Published
- 2010-06
- Volume
- 2
- Issue
- 2
- Pages
- 70-82
- ISSN
- 1943-0604 (ISSN-L)
- 1943-0612
- OpenAlex
- W2164424353
- Cited by
- 417
- References
- 55
- FWCI
- 22.45
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.