Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010)
The DOI matches its Crossref record; 125 of the 126 references it cites were found in a registry or an index.
Abstract
The simple, but general formal theory of fun and intrinsic motivation and creativity (1990-2010) is based on the concept of maximizing intrinsic reward for the active creation or discovery of novel, surprising patterns allowing for improved prediction or data compression. It generalizes the traditional field of active learning, and is related to old, but less formal ideas in aesthetics theory and developmental psychology. It has been argued that the theory explains many essential aspects of intelligence including autonomous development, science, art, music, and humor. This overview first describes theoretically optimal (but not necessarily practical) ways of implementing the basic computational principles on exploratory, intrinsically motivated agents or robots, encouraging them to provoke event sequences exhibiting previously unknown, but learnable algorithmic regularities. Emphasis is put on the importance of limited computational resources for online prediction and compression. Discrete and continuous time formulations are given. Previous practical, but nonoptimal implementations (1991, 1995, and 1997-2002) are reviewed, as well as several recent variants by others (2005-2010). A simplified typology addresses current confusion concerning the precise nature of intrinsic motivation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This underwrites Bayesian experimental design [134] and active learning in statistics [135], intrinsic motivation and artificial curiosity in machine learning and robotics [136, 137, 138, 139, 140].”
From The free energy principle made simpler but not too simple (2023) · cites this paper as [137]
In the citing paper
“This suggests that risk-averse policies minimise expected complexity or computational cost [137].”
From The free energy principle made simpler but not too simple (2023) · cites this paper as [137]
In the citing paper
“Ideally, the learner learns to choose actions that will lead to the largest improvements in its prediction or data compression performance (Schmidhuber, 2010).”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Schmidhuber, 2010
In the citing paper
“In Schmidhuber (2010), this is done in the framework of reinforcement learning, and incorporates a mechanisms for the system to measure its own rate of learning.”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Schmidhuber, 2010
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 7×
“Another potential drawback is the nature of the particular RNN-based RL method.”
BackgroundInfluentialSelf-citationA Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
Holland, J. H. (1992) · The MIT Press eBooks · doi:10.7551/mitpress/1090.001.0001 · Cited by 47,002 · cited 1×
“iii) An evolutionary algorithm [22], [29], [59],”
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 7×
“Some Q-learning accelerators simply update pairs of actions and states with currently quickly changing Q-values more frequently than others (that is, Q-values with high first derivatives are favored).”
BackgroundMethodInfluentialReinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Neural networks for pattern recognition
(1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,538
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972 · cited 4×
“, some typically suboptimal RL method [35].”
BackgroundMethodInfluential
Research works printed but not linked: 24
- kullback, 1959, Statistics and Information Theory
- hutter, 2004, Universal Artificial Intelligence Sequential Decisions Based on Algorithmic Probability
- holland, 1975, Adaptation in Natural and Artificial Systems
- schmidhuber, 2010, How to Learn Programs with Artificial Recurrent Neural Networks
- harlow, 1950, novelty and curiosity as determinants of exploratory behavior, J Exp Psychol, 41, 68
- berlyne, 1950, novelty and curiosity as determinants of exploratory behavior, Brit J Psychol, 41, 68
- 10.1007/bf00993591 · doi:10.1007/bf00993591 · Paper page
- levin, 1973, universal sequential search problems, Problems Inform Trans, 9, 265
- schmidhuber, 1991, Advances in Neural Information Processing Systems 3, nips 3, 500
- schmidhuber, 1997, Femmes Fractales
- schmidhuber, 2002, Advances in Evolutionary Computing, 579
- ring, 1994, Continual Learning in Reinforcement Environments
- schmidhuber, 1990, Dynamische neuronale Netze und das fundamentale raumzeitliche Lernproblem
- schmidhuber, 1991, Adaptive Curiosity and Adaptive Confidence
- schmidhuber, 2009, Compression Progress The Algorithmic Principle Behind Curiosity and Creativity (With Applications of the Theory of Humor) 40 min Video of Invited Talk at Singularity Summit 2009
- schmidhuber, 2009, Multiple Ways to Design Research Research Cases that Reshape the Design Discipline, 98
- singh, 2005, Advances in neural information processing systems, 17
- schwefel, 1974, Numerische Optimierung von Computer-Modellen
- schmidhuber, 2007, alle berechenbaren universen (all computable universes), Spektrum der Wissenschaft, 75
- storck, 1995, reinforcement driven information acquisition in non-deterministic environments, Proc Int Conf Artif Neural Netw, 2, 159
- brafman, 2002, R-MAX—A general polynomial time algorithm for near-optimal reinforcement learning, J Mach Learn Res, 3, 213
- frank, 2002, sthetische Information Estetika informacio Eine Einfhrung in die Kybernetische sthetik
- franke, 1979, Kybernetische sthetik Phnomen Kunst
- schmidhuber, 2006, Artificial General Intelligence, 175
Cited by
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815
“Note that ICM-pixels is representative of previous methods which compute information gain by directly using the observation space (Schmidhuber, 2010; Stadie et al., 2015).”
BackgroundMethodInfluentialAn Introduction to Deep Reinforcement Learning
Vincent, F.-L., Peter, H., Riashat, I., Marc, G. B., & Joelle, P. (2018) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000071 · Cited by 1,286
“Another common approach is to have a directed scheme thanks to exploration rewards given to the agent via heuristics that estimate novelty (Schmidhuber, 2010; Stadie et al., 2015; Houthooft et al., 2016).”
REBUS and the anarchic brain: Toward a unified model of the brain action of psychedelics
Carhart-Harris, R. L., & Friston, K. J. (2019) · Pharmacological Reviews · doi:10.1124/pr.118.017160 · Cited by 1,180
“, 2017) and is consistent with an explorative search strategy, which entails exploring novelty for the sake of significant knowledge gain (also known as intrinsic motivation in robotics) (Oudeyer and Kaplan, 2007; Schmidhuber, 2010; Barto et al., 2013).”
BackgroundMulti-agent deep reinforcement learning: a survey
Gronauer, S., & Diepold, K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-09996-w · Cited by 957
“One way to realize this is with intrinsic motivation, which is based on the concept of maximizing an internal reinforcement signal by actively discovering novel or surprising patterns (Chentanez et al. 2005; Oudeyer and Kaplan 2007; Schmidhuber 2010).”
BackgroundHierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Kulkarni, T. D., Narasimhan, K., Saeedi, A., & Tenenbaum, J. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06057 · Cited by 770
Predictive Processes and the Peculiar Case of Music
Koelsch, S., Vuust, P., & Friston, K. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.10.006 · Cited by 557
“Epistemic value: also known as intrinsic value [26,75,76] or the value of information [64,77].”
Noisy Networks for Exploration
Fortunato, M., Azar, M. G., Piot, B., Menick, J., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.10295 · Cited by 543
“Many such terms have been proposed, including learning progress [26], compression progress [30], variational information maximisation [14] and prediction gain [4].”
BackgroundExploration in deep reinforcement learning: A survey
Ladosz, P., Weng, L., Kim, M., & Oh, H. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.003 · Cited by 525
Lesort, T., Lomonaco, V., Stoian, A., Maltoni, D., Filliat, D., & Díaz-Rodríguez, N. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.004 · Cited by 506
Neural scene representation and rendering
Eslami, S. M. A., Jimenez Rezende, D., Besse, F., Viola, F., Morcos, A. S., Garnelo, M., Ruderman, A., Rusu, A. A., Danihelka, I., Gregor, K., et al. (2018) · Science · doi:10.1126/science.aar6170 · Cited by 495
VIME: Variational Information Maximizing Exploration
Houthooft, R., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09674 · Cited by 472
Model-Based Reinforcement Learning for Atari
Kaiser, L., Babaeizadeh, M., Milos, P., Osinski, B., Campbell, R. H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., et al. (2019) · arXiv · doi:10.48550/arxiv.1903.00374 · Cited by 467
“2017, Chapter 8), capturing important details of the scene (Ha & Schmidhuber, 2018), encouraging exploration (Oh et al., 2015), creating intrinsic motivation (Schmidhuber, 2010) or counterfactual reasoning (Buesing et al., 2019).”
BackgroundActive learning of inverse models with intrinsically motivated goal exploration in robots
Baranes, A., & Oudeyer, P.-Y. (2013) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2012.05.008 · Cited by 455
Recurrent World Models Facilitate Policy Evolution
Ha, D., & Schmidhuber, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.01999 · Cited by 429
“In particular, we can augment the reward function based on improvement in compression quality [77, 80, 81, 83].”
BackgroundSelf-citationLarge-Scale Study of Curiosity-Driven Learning
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., & Efros, A. A. (2018) · arXiv · doi:10.48550/arxiv.1808.04355 · Cited by 401
“Schmidhuber [37] and Oudeyer [25], Oudeyer and Kaplan [26] provide a great review of some of the earlier work on approaches to intrinsic motivation.”
BackgroundIn vitro neurons learn and exhibit sentience when embodied in a simulated game-world
Kagan, B. J., Kitchen, A. C., Tran, N. T., Habibollahi, F., Khajehnejad, M., Parker, B. J., Bhat, A., Rollo, B., Razi, A., & Friston, K. J. (2022) · Neuron · doi:10.1016/j.neuron.2022.09.001 · Cited by 400
Active Inference, Curiosity and Insight
Friston, K. J., Lin, M., Frith, C. D., Pezzulo, G., Hobson, J. A., & Ondobaka, S. (2017) · Neural Computation · doi:10.1162/neco_a_00999 · Cited by 351
“…expected accuracy and expected complexity, then the minimizing expected free energy entails the minimization of expected complexity (i.e., maximization of epistemic value), thereby furnishing more parsimonious, nicely compressed explanations for the nature of the sampled world (Schmidhuber, 2010).”
BackgroundTeacher–Student Curriculum Learning
Matiisen, T., Oliver, A., Cohen, T., & Schulman, J. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2934906 · Cited by 320
“Using learning progress as the reward can be linked to the concept of intrinsic motivation (Schmidhuber, 2010).”
BackgroundDynamics-Aware Unsupervised Discovery of SkillsPreprint
Sharma, A., Gu, S., Levine, S., Kumar, V., & Hausman, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.01657 · Cited by 362
“Another line of work that is conceptually close to our method copes with intrinsic motivation [51, 52, 58] which is used to drive the agent’s exploration.”
Method#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04717 · Cited by 296
Boosting Self-Supervised Learning via Knowledge Transfer
Noroozi, M., Vinjimoor, A., Favaro, P., & Pirsiavash, H. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00975 · Cited by 296
Gerstner, W., Lehmann, M., Liakoni, V., Corneil, D., & Brea, J. (2018) · Frontiers in Neural Circuits · doi:10.3389/fncir.2018.00053 · Cited by 290
“Theoreticians have studied these ideas in the context of curiosity (Schmidhuber, 2010), information gain during active exploration (Storck et al.”
BackgroundInfluentialConti, E., Madhavan, V., Such, F. P., Lehman, J., Stanley, K. O., & Clune, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.06560 · Cited by 108
“…a dynamics model that predicts future states (assuming predictions will be worse for rarely visited states/state-action pairs) (Stadie et al., 2015; Houthooft et al., 2016; Pathak et al., 2017), and (3) methods based on compression (novel states should be harder to compress) (Schmidhuber, 2010).”
BackgroundMethodVariational Information Maximisation for Intrinsically Motivated Reinforcement Learning
Mohamed, S., & Rezende, D. J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.08731 · Cited by 258
“…or Bayesian surprise, which uses the KL divergence to measure the change in an agents internal belief after the observation of new data [8, 22]; measures based on prediction errors of future states such predicted L 1 change, predicted mode change or probability gain [16], or salient event…”
BackgroundDeep Learning for Video Game Playing
Justesen, N., Bontrager, P., Togelius, J., & Risi, S. (2020) · IEEE Transactions on Games · doi:10.1109/tg.2019.2896986 · Cited by 257
Reinforcement Learning with Unsupervised Auxiliary Tasks
Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05397 · Cited by 247
Deeply Felt Affect: The Emergence of Valence in Deep Active Inference
Hesp, C., Smith, R., Parr, T., Allen, M., Friston, K. J., & Ramstead, M. J. D. (2021) · Neural Computation · doi:10.1162/neco_a_01341 · Cited by 246
“Furthermore, it generalizes many established formulations of optimal behavior (Itti & Baldi, 2009; Schmidhuber, 2010; Mirza, Adams, Mathys, & Friston, 2016; Veale, Hafed, & Yoshida, 2017) and provides a formal description of the motivated and self-preserving behavior of living systems (Friston,…”
BackgroundEfficient Exploration via State Marginal MatchingPreprint
Lee, L., Eysenbach, B., Parisotto, E., Xing, E. P., Levine, S., & Salakhutdinov, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05274 · Cited by 281
“Another set of approaches (Tang et al., 2017; Bellemare et al., 2016; Schmidhuber, 2010) directly encourage the agent to visit novel states.”
BackgroundSocial Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
Jaques, N., Lazaridou, A., Hughes, E., Gülçehre, Ç., Ortega, P. A., Strouse, D., Leibo, J. Z., & de Freitas, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.08647 · Cited by 241
“agent to learn across a variety of tasks (Singh et al., 2004). Previous approaches to intrinsic motivation have broadlyfallenintotwocategories: (1)curiosity,oradrivefornovelty(e.g. Pathaketal.(2017);Schmidhuber (2010)), and (2) empowerment, or a drive to be able to manipulate the environment (Klyubin et al., 2005). We…”
BackgroundActive inference on discrete state-spaces: A synthesis
Da Costa, L., Parr, T., Sajid, N., Veselic, S., Neacsu, V., & Friston, K. (2020) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2020.102447 · Cited by 226
“Intrinsic value (parameters) or novelty (16) Extrinsic value is just the expected value of log evidence, which can be associated with reward and utility in behavioural psychology and economics, respectively [144–146].”
BackgroundA free energy principle for a particular physics
Friston, K. (2019) · arXiv · doi:10.48550/arxiv.1906.10184 · Cited by 222
“Information gain is sometimes referred to as epistemic value or intrinsic motivation in artificial intelligence and robotics (Friston et al., 2015b; Oudeyer and Kaplan, 2007; Schmidhuber, 2010).”
BackgroundInfluentialAutomatic Goal Generation for Reinforcement Learning Agents
Florensa, C., Held, D., Geng, X., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06366 · Cited by 138
“Intrinsic Motivation: Intrinsic motivation involves learning with an intrinsically specified objec-tive (Schmidhuber, 1991, 2010).”
BackgroundLee, K., Smith, L., & Abbeel, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05091 · Cited by 284
“Instead, our insight is to produce informative queries at the start of training by utilizing unsupervised pre-training to collect diverse samples solely through intrinsic motivation (Oudeyer et al., 2007; Schmidhuber, 2010).”
BackgroundGregor, K., Rezende, D. J., & Wierstra, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07507 · Cited by 202
Deep Reinforcement LearningPreprint
Li, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.06339 · Cited by 214
Modern Bayesian Experimental Design
Rainforth, T., Foster, A., Ivanova, D. R., & Smith, F. B. (2024) · Statistical Science · doi:10.1214/23-sts915 · Cited by 372
Reconciling novelty and complexity through a rational analysis of curiosity.
Dubey, R., & Griffiths, T. L. (2020) · Psychological Review · doi:10.1037/rev0000175 · Cited by 192
Li, C. (2019) · Reinforcement Learning for Cyber-Physical Systems · doi:10.1201/9781351006620-6 · Cited by 188
Ha, D., & Schmidhuber, J. (2018) · Zenodo · doi:10.5281/zenodo.1207631 · Cited by 187
“Large RNNs are highly expressive models that can learn rich spatial and temporal representations of data.”
BackgroundMethodResultInfluentialSelf-citationSentience and the Origins of Consciousness: From Cartesian Duality to Markovian Monism
Friston, K. J., Wiese, W., & Hobson, J. A. (2020) · Entropy · doi:10.3390/e22050516 · Cited by 186
“, expected surprisal or uncertainty), we now have a plausible description of intentional behaviour that will, to all intents and purposes, look like uncertainty resolving, information seeking, epistemic foraging [81,83,84,26,85,86,87,88,89,90].”
BackgroundIn the body's eye: the computational anatomy of interoceptive inference
Allen, M., Levy, A., Parr, T., & Friston, K. J. (2022) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1010490 · Cited by 185
Hybrid Reward Architecture for Reinforcement Learning
van Seijen, H., Fatemi, M., Romoff, J., Laroche, R., Barnes, T., & Tsang, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.04208 · Cited by 176
“Intrinsic motivation (Stout et al., 2005; Schmidhuber, 2010) uses this observation to improve learning in sparse-reward domains, by adding a domain-specific intrinsic reward signal to the reward coming from the environment.”
BackgroundEnd-to-End Navigation Strategy With Deep Reinforcement Learning for Mobile Robots
Shi, H., Shi, L., Xu, M., & Hwang, K.-S. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2936167 · Cited by 176
“Recent studies have proposed various methods to obtain an intrinsic reward, including Bayesian surprise [22], information gain [23], measures by novelty of observed states [24], and measures by the agent’s learning progress in predicting action outcomes [25], [26].”
MethodComputational psychiatry: from synapses to sentience
Friston, K. J. (2022) · Molecular Psychiatry · doi:10.1038/s41380-022-01743-z · Cited by 197
Regimes of Expectations: An Active Inference Model of Social Conformity and Human Decision Making
Constant, A., Ramstead, M. J. D., Veissière, S. P. L., & Friston, K. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00679 · Cited by 171
“The latter can also be construed as the epistemic affordance that underwrites curiosity and novelty seeking (Schmidhuber, 2010; Friston et al., 2015; Friston et al., 2017b; Kaplan and Friston, 2018).”
BackgroundInfluentialParr, T., & Friston, K. J. (2019) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2018.10.006 · Cited by 168
“Salience has been defined (equivalently) in terms of intrinsic motivation [49,50], Bayesian surprise [51], a relative entropy [3],”
BackgroundBadcock, P. B., Friston, K. J., Ramstead, M. J. D., Ploeger, A., & Hohwy, J. (2019) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-019-00721-3 · Cited by 167
The free energy principle made simpler but not too simple
Friston, K., Da Costa, L., Sajid, N., Heins, C., Ueltzhöffer, K., Pavliotis, G. A., & Parr, T. (2023) · Physics Reports · doi:10.1016/j.physrep.2023.07.001 · Cited by 163
“This suggests that risk-averse policies minimise expected complexity or computational cost [137].”
Moulin-Frier, C., Nguyen, S. M., & Oudeyer, P.-Y. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.01006 · Cited by 162
Friston, K., Da Costa, L., Hafner, D., Hesp, C., & Parr, T. (2021) · Neural Computation · doi:10.1162/neco_a_01351 · Cited by 117
“Heuristically, this curiosity resolves uncertainty about “what would happen if I did that?” (Schmidhuber, 2010).”
BackgroundInfluentialNeural SLAM: Learning to Explore with External Memory
Zhang, J., Tai, L., Liu, M., Boedecker, J., & Burgard, W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.09520 · Cited by 101
A literature review on the influence of Kahoot! On learning outcomes, interaction, and collaboration
Zhang, Q., & Yu, Z. (2021) · Education and Information Technologies · doi:10.1007/s10639-021-10459-6 · Cited by 183
“However, excessive information with totally no predictability probably leads to incomprehension (Schmidhuber, 2010).”
BackgroundSymbol emergence in robotics: a survey
Taniguchi, T., Nagai, T., Nakamura, T., Iwahashi, N., Ogata, T., & Asoh, H. (2016) · Advanced Robotics · doi:10.1080/01691864.2016.1164622 · Cited by 150
Limits of End-to-End LearningPreprint
Glasmachers, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.08305 · Cited by 194
“It motivated various proposals that were popularized in the domains of control, reinforcement learning, and artificial intelligence [19, 1, 20, 21].”
BackgroundThe feeling of grip: novelty, error dynamics, and the predictive brain
Kiverstein, J., Miller, M., & Rietveld, E. (2019) · Synthese · doi:10.1007/s11229-017-1583-9 · Cited by 143
“A similar line of thinking comes from recent research on error reduction dynamics in artificial intelligence and robotics (Oudeyer et al. 2007, 2013; Schmidhuber 2010).”
MethodVariational ecology and the physics of sentient systems
Ramstead, M. J. D., Constant, A., Badcock, P. B., & Friston, K. J. (2019) · Physics of Life Reviews · doi:10.1016/j.plrev.2018.12.002 · Cited by 141
“On this view, the landscape of affordances is a product of inference about “what would happen if I did that?” [79].”
BackgroundOn Learning Intrinsic Rewards for Policy Gradient MethodsPreprint
Zheng, Z., Oh, J., & Singh, S. P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.06459 · Cited by 196
“In these and other similar examples [Schmidhuber, 2010, Stadie et al., 2015, Oudeyer and Kaplan, 2009], the agent’s learning performance improves through the reward transformations, but the reward transformations are expert-designed and not learned.”
BackgroundSlow Down to Go Better: A Survey on Slow Feature Analysis
Song, P., & Zhao, C. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3201621 · Cited by 148
“They proposed a curiosity-driven modular IncSFA based on the theory of artificial curiosity [62], [63] and reinforcement learning to prevent forgetting.”
BackgroundFun and games in higher education: an analysis of UK student perspectives
Whitton, N., & Langan, M. (2018) · Teaching in Higher Education · doi:10.1080/13562517.2018.1541885 · Cited by 148
“While from a computer modelling perspective, Schmidhuber (2010) describes fun as the internal joy for the discovery of creation of novel patterns, where a pattern is interesting or surprising.”
BackgroundPark, J., Seo, Y., Shin, J., Lee, H., Abbeel, P., & Lee, K. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.10050 · Cited by 146
Schmidhuber, J. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00313 · Cited by 118
Self-citationPlanning to be surprised: optimal bayesian exploration in dynamic environments
Sun, Y., Gomez, F., & Schmidhuber, J. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22887-2_5 · Cited by 134
“Conceptually, however, this work is closely connected to artificial curiosity and intrinsically motivated reinforcement learning [6,7,9] for agents that actively explore the environment without an external reward signal.”
BackgroundSelf-citationRIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated EnvironmentsPreprint
Răileanu, R., & Rocktäschel, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12292 · Cited by 207
“Inspired by human learning, the use of intrinsic motivation has been proposed to encourage agents to learn about their environments even when extrinsic feedback is rarely provided (Schmidhuber, 1991b; 2010; Oudeyer et al., 2007; Oudeyer & Kaplan, 2009).”
BackgroundHierarchical Reinforcement Learning: A Survey and Open Research Challenges
Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4010009 · Cited by 129
“Hierarchical reinforcement learning (HRL) often utilizes various forms of intrinsic motivation [49] in order to provide additional denser reward signals for individual abstractions.”
BackgroundMaking Sense of the World: Infant Learning From a Predictive Processing Perspective
Köster, M., Kayhan, E., Langeloh, M., & Hoehl, S. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691619895071 · Cited by 111
“In developmental robotics, the imperative to learn about one’s own motor capacities is often cast in terms of an intrinsic motivation (see also the concept of motor babbling; Barto, Mirolli, & Baldassarre, 2013; Friston et al., 2015, Kaplan & Oudeyer, 2007; Schmidhuber, 2010).”
BackgroundDeep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey
Majid, A. Y., Saaybi, S., van Rietbergen, T., François-Lavet, V., Prasad, R. R. V., & Verhoeven, C. J. M. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3264540 · Cited by 143
Schmidhuber, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.09249 · Cited by 138
Self-citationDark control: The default mode network as a reinforcement learning agent
Dohmatob, E., Dumas, G., & Bzdok, D. (2020) · Human Brain Mapping · doi:10.1002/hbm.25019 · Cited by 121
Humans monitor learning progress in curiosity-driven exploration
Ten, A., Kaushik, P., Oudeyer, P.-Y., & Gottlieb, J. (2021) · Nature Communications · doi:10.1038/s41467-021-26196-w · Cited by 94
“However, if the environment is replete with challenging and unlearnable tasks, e.g., during realistic scienti fi c investigation, an LP-based strategy may be more critical for steering learners toward tasks where progress is made as proposed in arti fi cial curiosity 25,26,29 .”
BackgroundToward an integration of deep learning and neuroscience
Marblestone, A. H., Wayne, G., & Kording, K. P. (2016) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2016.00094 · Cited by 119
An Active Inference Approach to Modeling Structure Learning: Concept Learning as an Example Case
Smith, R., Schwartenbeck, P., Parr, T., & Friston, K. J. (2020) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2020.00041 · Cited by 99
Watters, N., Matthey, L., Bošnjak, M., Burgess, C., & Lerchner, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09275 · Cited by 64
“Curiosity-driven exploration was first proposed in (Schmidhuber, 1990b, 1991) and expanded upon in (Schmidhuber, 2010).”
BackgroundA World Unto Itself: Human Communication as Active Inference
Vasil, J., Badcock, P. B., Constant, A., Friston, K., & Ramstead, M. J. D. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.00417 · Cited by 94
“See MacKay (1995), Wallace and Dowe (1999), and Schmidhuber (2010) for more discussion.”
BackgroundWhat are intrinsic motivations? A biological perspective
Baldassarre, G. (2011) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2011.6037367 · Cited by 135
LLM Fine-Tuning: Concepts, Opportunities, and Challenges
Wu, X.-K., Chen, M., Li, W., Wang, R., Lu, L., Liu, J., Hwang, K., Hao, Y., Pan, Y., Meng, Q., et al. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9040087 · Cited by 154
“These ideas have been later interpreted as involving concepts like “autonomous thinking” [2], “self-learning” [3], and “self-evolving” [4].”
The Feeling of Action Tendencies: On the Emotional Regulation of Goal-Directed Behavior
Lowe, R., & Ziemke, T. (2011) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00346 · Cited by 99
Intelligent problem-solving as integrated hierarchical reinforcement learning
Eppe, M., Gumbsch, C., Kerzel, M., Nguyen, P. D. H., Butz, M. V., & Wermter, S. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-021-00433-9 · Cited by 140
Linson, A., Clark, A., Ramamoorthy, S., & Friston, K. (2018) · Frontiers in Robotics and AI · doi:10.3389/frobt.2018.00021 · Cited by 90
“This intrinsic value of a particular policy or model appears in many guises, most notably as intrinsic motivation in robotics (Oudeyer and Kaplan, 2007; Schmidhuber, 2010), the value of information in economics (Howard, 1966), and Bayesian surprise in models of exploration and visual searches (Schmidhuber, 1991; Itti…”
BackgroundRepresentation Wars: Enacting an Armistice Through Active Inference
Constant, A., Clark, A., & Friston, K. J. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.598733 · Cited by 95
“As such, it is sometimes referred to as intrinsic motivation (Ryan and Deci, 1985; Oudeyer and Kaplan, 2007; Schmidhuber, 2010; Barto et al., 2013); Schmidhuber, 2010).”
BackgroundNave, K., Deane, G., Miller, M., & Clark, A. (2020) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.1542 · Cited by 110
BYOL-Explore: Exploration by Bootstrapped Prediction
Zhaohan, G., Thakoor, S., Pislar, M., Pires, B. Á., Altché, F., Tallec, C., Saade, A., Calandriello, D., Grill, J.-B., Tang, Y., et al. (2022) · neural information processing systems · doi:10.52202/068431-2309 · Cited by 103
World model learning and inference
Friston, K., Moran, R. J., Nagai, Y., Taniguchi, T., Gomi, H., & Tenenbaum, J. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.09.011 · Cited by 90
“Optimising free energy, therefore, puts pressure on finding the simplest explanations and models (Schmidhuber, 2010).”
BackgroundAffective Value in the Predictive Mind
Van De Cruys, S. (2017) · Theoretical Philosophy/MIND Group – JGU Mainz · doi:10.15502/9783958573253 · Cited by 100
“Predictive progress has already been used to understand and implement intrinsic rewards in the domain of artificial intelligence (Kaplan and Oudeyer 2007; Schmidhuber 2010).”
BackgroundGRAIL: A Goal-Discovering Robotic Architecture for Intrinsically-Motivated Learning
Santucci, V. G., Baldassarre, G., & Mirolli, M. (2016) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2016.2538961 · Cited by 94
“as intrinsic reinforcements based on some form of “information compression progress” [80], in particular the prediction error (PE), or the improvement in the PE (PEI), of a predictor”
BackgroundIs Pessimism Provably Efficient for Offline Reinforcement Learning?
Jin, Y., Yang, Z., & Wang, Z. (2024) · Mathematics of Operations Research · doi:10.1287/moor.2022.0216 · Cited by 103
Computer-inspired quantum experiments
Krenn, M., Erhard, M., & Zeilinger, A. (2020) · Nature Reviews Physics · doi:10.1038/s42254-020-0230-4 · Cited by 86
Play, Curiosity, and Cognition
Chu, J., & Schulz, L. E. (2020) · Annual Review of Developmental Psychology · doi:10.1146/annurev-devpsych-070120-014806 · Cited by 93
“Stopping for knowledge”: The sense of beauty in the perception-action cycle
Sarasso, P., Neppi-Mòdona, M., Sacco, K., & Ronga, I. (2020) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.09.004 · Cited by 97
Towards a Definition of Generative Artificial Intelligence
Ronge, R., Maier, M., & Rathgeber, B. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00863-y · Cited by 101
“In this realm of possibilities and potentials, GenAI is a unique and novel mode of operation and clearly detached from earlier reductionist misconceptions about machine creativity (Schmidhuber, 2010).”
Bayesian model reductionPreprint
Friston, K. J., Parr, T., & Zeidman, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.07092 · Cited by 104
“Over the past few years, Bayesian model comparison and structure learning have become key issues in the neurosciences [1-6]; particularly, in optimising models of neuroimaging timeseries [7, 8] – and as a fundamental problem that our brains have to solve [9-11].”
BackgroundCuriosity as a Metacognitive Feeling
Goupil, L., & Proust, J. (2023) · Cognition · doi:10.1016/j.cognition.2022.105325 · Cited by 89
Tulver, K., Kaup, K., Laukkonen, R., & Aru, J. (2023) · Consciousness and Cognition · doi:10.1016/j.concog.2023.103494 · Cited by 97
Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
Li, R., Jabri, A., Darrell, T., & Agrawal, P. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9197468 · Cited by 78
“using better optimization methods [9], [10], [11], combining model-based and model-free learning [12], hierarchical learning [13], and design of better exploration methods [14], [15], [16], [17], [18].”
MethodCreativity and Machine Learning: A Survey
Franceschelli, G., & Musolesi, M. (2024) · ACM Computing Surveys · doi:10.1145/3664595 · Cited by 103
“Finally, in [101] Schmidhuber formalizes his theory about creativity and his way to achieve it by using reinforcement learning (see also [99] or [100]).”
BackgroundMethodDesigning ecosystems of intelligence from first principles
Friston, K. J., Ramstead, M. J. D., Kiefer, A. B., Tschantz, A., Buckley, C. L., Albarracin, M., Pitliya, R. J., Heins, C., Klein, B., Millidge, B., et al. (2024) · Collective Intelligence · doi:10.1177/26339137231222481 · Cited by 94
“Simulated neural networks, for example, often over fi t and fail to generalize if they are not forced to learn compressed representations of their inputs (MacKay, 1995; Olshausen and Field, 1996; Schmidhuber, 2010; Wallace and Dowe, 1999).”
BackgroundClune, J. (2019) · arXiv · doi:10.48550/arxiv.1905.10985 · Cited by 82
“There have been many other important recent papers investigating how much agents can learn when motivated by curiosity alone [19, 128, 126, 147, 42, 56, 127, 150].”
BackgroundA computational model of aesthetic value.
Brielmann, A. A., & Dayan, P. (2022) · Psychological Review · doi:10.1037/rev0000337 · Cited by 87
Insight and the selection of ideas
Laukkonen, R., Webb, M. E., Salvi, C., Tangen, J. M., Slagter, H. A., & Schooler, J. W. (2023) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105363 · Cited by 87
“In some fields, this kind of (intrinsic) value is known as intrinsic motivation (Oudeyer et al., 2007; Schmidhuber, 2010; Schwartenbeck et al., 2019), of the sort that underwrites exploration and epistemic affordances.”
Improving Intrinsic Exploration with Language Abstractions
Mu, J., Zhong, V., Răileanu, R., Jiang, M., Goodman, N. D., Rocktäschel, T., & Grefenstette, E. (2022) · neural information processing systems · doi:10.52202/068431-2460 · Cited by 80
Unveiling the Mind of the Machine
Clegg, M., Hofstetter, R., de Bellis, E., & Schmitt, B. H. (2023) · Journal of Consumer Research · doi:10.1093/jcr/ucad075 · Cited by 85
Metadata
- Authors
- Schmidhuber, J.
- Affiliations
- Dalle Molle Institute for Artificial Intelligence Research (CH)
- Corresponding author
- Schmidhuber, Jürgen
- ORCID
- Schmidhuber, Jürgen
- Venue
- IEEE Transactions on Autonomous Mental Development
- Published
- 2010-09
- Volume
- 2
- Issue
- 3
- Pages
- 230-247
- ISSN
- 1943-0604 (ISSN-L)
- 1943-0612
- OpenAlex
- W2034806191
- Cited by
- 747
- References
- 126
- Citations by year
- 2026: 29 · 2025: 64 · 2024: 46 · 2023: 50 · 2022: 60 · 2021: 69 · 2020: 92 · 2019: 72 · 2018: 55 · 2017: 37 · 2016: 26 · 2015: 24 · 2014: 30 · 2013: 28 · 2012: 41
- FWCI
- 31.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Autonomous Mental Development (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Creativity in Education and Neuroscience
- Keywords
- intrinsic motivation, creativity, fun, intrinsic rewards, active learning (machine learning), novelty seeking, surprise, pattern discovery, data compression, online prediction, autonomous agents, robots, aesthetics, humor, intelligence, limited computational resources, continuous-time formulation, discrete-time formulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.