The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Intrinsically Motivated Reinforcement Learning: An Evolutionary Perspective
Abstract
There is great interest in building intrinsic motivation into artificial systems using the reinforcement learning framework. Yet, what intrinsic motivation may mean computationally, and how it may differ from extrinsic motivation, remains a murky and controversial subject. In this paper, we adopt an evolutionary perspective and define a new optimal reward framework that captures the pressure to design good primary reward functions that lead to evolutionary success across environments. The results of two computational experiments show that optimal primary reward signals may yield both emergent intrinsic and extrinsic motivation. The evolutionary perspective and the associated optimal reward framework thus lead to the conclusion that there are no hard and fast features distinguishing intrinsic and extrinsic reward computationally. Rather, the directness of the relationship between rewarding behavior and evolutionary success varies along a continuum.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Self-citationIntrinsic Motivation and Self-Determination in Human Behavior
Deci, E. L., & Ryan, R. M. (1985) · doi:10.1007/978-1-4899-2271-7 · Cited by 26,772
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions
Ryan, R. M., & Deci, E. L. (2000) · Contemporary Educational Psychology · doi:10.1006/ceps.1999.1020 · Cited by 18,666
Intrinsic Motivation and Self-Determination in Human Behavior
Miller, K. A., Deci, E. L., & Ryan, R. M. (1988) · Contemporary Sociology · doi:10.2307/2070638 · Cited by 15,312
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Skinner, B. F., & Hull, C. L. (1944) · The American Journal of Psychology · doi:10.2307/1416955 · Cited by 4,051
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1023/a:1022676722315 · Cited by 3,907
Skinner, B. F. (1938) · Cited by 3,571
Neuronlike adaptive elements that can solve difficult learning control problems
Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1983.6313077 · Cited by 3,233
Self-citationOlds, J., & Milner, P. (1954) · Journal of Comparative and Physiological Psychology · doi:10.1037/h0058775 · Cited by 3,176
Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping
Ng, A. Y., Harada, D., & Russell, S. (1999) · Cited by 1,744
Transfer Learning for Reinforcement Learning Domains: A Survey
Taylor, M. E., & Stone, P. (2009) · doi:10.5555/1577069.1755839 · Cited by 1,488
Sutton, R. S. (1990) · Machine Learning Proceedings 1990 · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,314
Berlyne, D. E. (1966) · Science · doi:10.1126/science.153.3731.25 · Cited by 1,137
Intrinsic Motivation Systems for Autonomous Mental Development
Oudeyer, P.-Y., Kaplan, F., & Hafner, V. V. (2007) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2006.890271 · Cited by 1,126
Recent Advances in Hierarchical Reinforcement Learning
Barto, A. G., & Mahadevan, S. (2003) · Discrete Event Dynamic Systems · doi:10.1023/a:1022140919877 · Cited by 1,024
Self-citationWhat is intrinsic motivation? A typology of computational approaches
Oudeyer, P.-Y. (2007) · Frontiers in Neurorobotics · doi:10.3389/neuro.12.006.2007 · Cited by 811
Motivation: theory and research
Cofer, C. N., & Appley, M. H. (1964) · Medical Entomology and Zoology · Cited by 622
Intrinsically Motivated Reinforcement Learning
Singh, S., Barto, A. G., & Chentanez, N. (2005) · doi:10.21236/ada440280 · Cited by 610
Self-citationA computational substrate for incentive salience
McClure, S. M., Daw, N. D., & Read Montague, P. (2003) · Trends in Neurosciences · doi:10.1016/s0166-2236(03)00177-2 · Cited by 422
A Possibility for Implementing Curiosity and Boredom in Model-Building Neural Controllers
Schmidhuber, J. (1991) · From Animals to Animats · doi:10.7551/mitpress/3115.003.0030 · Cited by 421
A Behavior System: An Introduction to Behavior Theory concerning the Individual Organism
Leeper, R., & Hull, C. L. (1954) · The American Journal of Psychology · doi:10.2307/1418653 · Cited by 392
Motivation: Theory and Research
Silverman, S. (1965) · American Journal of Psychotherapy · doi:10.1176/appi.psychotherapy.1965.19.4.682 · Cited by 375
Intrinsically Motivated Learning of Hierarchical Collections of Skills
Barto, A. G., Singh, S. P., & Chentanez, N. (2004) · Cited by 374
Self-citationLeeper, R., & Hull, C. L. (1952) · The American Journal of Psychology · doi:10.2307/1418777 · Cited by 374
Arita, T., & Suzuki, R. (2000) · Artificial Life VII · doi:10.7551/mitpress/1432.003.0027 · Cited by 353
Learning to Drive a Bicycle Using Reinforcement Learning and Shaping
Randløv, J., & Alstrøm, P. (1998) · Cited by 324
Recent Advances in Hierarchical Reinforcement Learning
Barto, A. G., & Mahadevan, S. (2003) · Discrete Event Dynamic Systems · doi:10.1023/a:1025696116075 · Cited by 317
Self-citationLearning and satiation of response in intrinsically motivated complex puzzle performance by monkeys.
Harlow, H. F. (1950) · Journal of Comparative and Physiological Psychology · doi:10.1037/h0058114 · Cited by 287
AM, an artificial intelligence approach to discovery in mathematics as heuristic search
Lenat, D. B. (1976) · Cited by 255
Autonomous shaping: knowledge transfer in reinforcement learning
Konidaris, G., & Barto, A. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143906 · Cited by 184
Self-citationIntrinsically Motivated Hierarchical Skill Learning in Structured Environments
Vigorito, C. M., & Barto, A. G. (2010) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2010.2050205 · Cited by 110
Self-citationBarto, A. G., Lewis, R. L., & Singh, S. P. (2009) · eScholarship (California Digital Library) · Cited by 96
Self-citationEvolving internal reinforcers for an intrinsically motivated reinforcement-learning robot
Schembri, M., Mirolli, M., & Baldassarre, G. (2007) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2007.4354052 · Cited by 87
Transfer in variable-reward hierarchical reinforcement learning
Mehta, N., Natarajan, S., Tadepalli, P. V., & Fern, A. (2008) · Machine Learning · doi:10.1007/s10994-008-5061-y · Cited by 84
Motivated Reinforcement Learning
Merrick, K. E., & Maher, M. L. (2009) · Springer eBooks · doi:10.1007/978-3-540-89187-1 · Cited by 74
On the role of tracking in stationary environments
Sutton, R. S., Koop, A., & Silver, D. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273606 · Cited by 70
Information, evolution and utilityPreprint
Samuelson, L., & Samuelson, L. (2006) · RePEc: Research Papers in Economics · Cited by 65
Food-seeking drive, affective process, and learning.
Young, P. T. (1949) · Psychological Review · doi:10.1037/h0060312 · Cited by 63
Constrained reinforcement learning from intrinsic and extrinsic rewards
Uchibe, E., & Doya, K. (2007) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2007.4354030 · Cited by 48
Schmidhuber, J. (1997) · SUPSI ARIS · doi:10.71910/supsi.7455 · Cited by 44
Artificial curiosity based on discovering novel algorithmic predictability through coevolution
Schmidhuber, J. (2003) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec.1999.785467 · Cited by 42
Co-evolution of Shaping Rewards and Meta-Parameters in Reinforcement Learning
Elfwing, S., Uchibe, E., Doya, K., & Christensen, H. I. (2008) · Adaptive Behavior · doi:10.1177/1059712308092835 · Cited by 36
Finding intrinsic rewards by embodied evolution and constrained reinforcement learning
Uchibe, E., & Doya, K. (2008) · Neural Networks · doi:10.1016/j.neunet.2008.09.013 · Cited by 23
Adaptation in Constant Utility Non-Stationary Environments.
Littman, M. L., & Ackley, D. H. (1991) · ICGA · Cited by 22
Schultz, W. (2007) · Scholarpedia · doi:10.4249/scholarpedia.1652 · Cited by 21
Artificial motives: A review of motivation in artificial creatures
Savage, T. (2000) · Connection Science · doi:10.1080/095400900750060131 · Cited by 18
Schultz, W. (2007) · Scholarpedia · doi:10.4249/scholarpedia.2184 · Cited by 15
Evolution of Valence Systems in an Unstable Environment
Snel, M., & Hayes, G. M. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-69134-1_2 · Cited by 8
Valency for Adaptive Homeostatic Agents: Relating Evolution and Learning
Damoulas, T., Cos, I., Hayes, G. M., & Taylor, T. (2005) · Lecture notes in computer science · doi:10.1007/11553090_94 · Cited by 8
Transfer Learning in Reinforcement Learning
Yang, Q., Zhang, Y., Dai, W., & Pan, S. J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781139061773.010 · Cited by 3
Cited by
Neuronal Reward and Decision Signals: From Theories to Data
Schultz, W. (2015) · Physiological Reviews · doi:10.1152/physrev.00023.2014 · Cited by 1,126
Information-seeking, curiosity, and attention: computational and neural mechanisms
Gottlieb, J., Oudeyer, P.-Y., Lopes, M., & Baranes, A. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.09.001 · Cited by 784
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Kulkarni, T. D., Narasimhan, K., Saeedi, A., & Tenenbaum, J. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06057 · Cited by 770
Human-level performance in 3D multiplayer games with population-based reinforcement learning
Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L., Lever, G., Castañeda, A. G., Beattie, C., Rabinowitz, N. C., Morcos, A. S., Ruderman, A., et al. (2019) · Science · doi:10.1126/science.aau6249 · Cited by 662
Tamar, A., Wu, Y., Thomas, G., Levine, S., & Abbeel, P. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/700 · Cited by 484
Metadata
- Authors
- Singh, S., Lewis, R. L., Barto, A. G., & Sorg, J.
- Venue
- IEEE Transactions on Autonomous Mental Development
- Published
- 2010-06
- Volume
- 2
- Issue
- 2
- Pages
- 70-82
- ISSN
- 1943-0604 (ISSN-L)
- 1943-0612
- OpenAlex
- W2164424353
- Cited by
- 417
- References
- 55
- FWCI
- 22.45
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.