Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010)
The DOI matches its Crossref record; 125 of the 126 references it cites were found in a registry or an index.
Abstract
The simple, but general formal theory of fun and intrinsic motivation and creativity (1990-2010) is based on the concept of maximizing intrinsic reward for the active creation or discovery of novel, surprising patterns allowing for improved prediction or data compression. It generalizes the traditional field of active learning, and is related to old, but less formal ideas in aesthetics theory and developmental psychology. It has been argued that the theory explains many essential aspects of intelligence including autonomous development, science, art, music, and humor. This overview first describes theoretically optimal (but not necessarily practical) ways of implementing the basic computational principles on exploratory, intrinsically motivated agents or robots, encouraging them to provoke event sequences exhibiting previously unknown, but learnable algorithmic regularities. Emphasis is put on the importance of limited computational resources for online prediction and compression. Discrete and continuous time formulations are given. Previous practical, but nonoptimal implementations (1991, 1995, and 1997-2002) are reviewed, as well as several recent variants by others (2005-2010). A simplified typology addresses current confusion concerning the precise nature of intrinsic motivation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This underwrites Bayesian experimental design [134] and active learning in statistics [135], intrinsic motivation and artificial curiosity in machine learning and robotics [136, 137, 138, 139, 140].”
From The free energy principle made simpler but not too simple (2023) · cites this paper as [137]
In the citing paper
“This suggests that risk-averse policies minimise expected complexity or computational cost [137].”
From The free energy principle made simpler but not too simple (2023) · cites this paper as [137]
In the citing paper
“Ideally, the learner learns to choose actions that will lead to the largest improvements in its prediction or data compression performance (Schmidhuber, 2010).”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Schmidhuber, 2010
In the citing paper
“In Schmidhuber (2010), this is done in the framework of reinforcement learning, and incorporates a mechanisms for the system to measure its own rate of learning.”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Schmidhuber, 2010
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 7×
“Another potential drawback is the nature of the particular RNN-based RL method.”
BackgroundInfluentialSelf-citationA Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
Holland, J. H. (1992) · The MIT Press eBooks · doi:10.7551/mitpress/1090.001.0001 · Cited by 47,002 · cited 1×
“iii) An evolutionary algorithm [22], [29], [59],”
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 7×
“Some Q-learning accelerators simply update pairs of actions and states with currently quickly changing Q-values more frequently than others (that is, Q-values with high first derivatives are favored).”
BackgroundMethodInfluentialReinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Neural networks for pattern recognition
(1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,538
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972 · cited 4×
“, some typically suboptimal RL method [35].”
BackgroundMethodInfluentialA Method for the Construction of Minimum-Redundancy Codes
Huffman, D. (1952) · Proceedings of the IRE · doi:10.1109/jrproc.1952.273898 · Cited by 6,334
Modeling by shortest data description
Rissanen, J. (1978) · Automatica · doi:10.1016/0005-1098(78)90005-5 · Cited by 6,011 · cited 3×
“Remarkably, under very general conditions both universal inductive inference [45], [109], [110] and the compression-oriented MDL approach [37], [45], [62], [115], [116] converge to the correct predictions in the limit [56].”
BackgroundConflict, arousal, and curiosity.
Berlyne, D. E. (1960) · McGraw-Hill Book Company · doi:10.1037/11164-000 · Cited by 5,995
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1023/a:1022676722315 · Cited by 3,907 · cited 6×
“Such introspective abilities are important to predict future intrinsic rewards which depend on the already existing knowledge encoded in the predictor. c) Which optimizer of expected intrinsic and extrinsic reward is used? i) A traditional Q-learner [117] able to deal with delayed rewards as long as the environment is…”
BackgroundMethodInfluentialAn Introduction to Kolmogorov Complexity and Its Applications
Li, M., & Vitányi, P. (2019) · Texts in Computer Science · doi:10.1007/978-3-030-11298-1 · Cited by 3,681
Evolutionsstrategie : Optimierung technischer Systeme nach Prinzipien der biologischen Evolution
Rechenberg, I. (1973) · Frommann-Holzboog eBooks · Cited by 2,440
Fedorov, V. V., Studden, W. J., & Klimko, E. M. (1972) · Cited by 2,092
Three approaches to the quantitative definition of information*
Kolmogorov, A. N. (1968) · International Journal of Computer Mathematics · doi:10.1080/00207166808803030 · Cited by 2,031 · cited 10×
“The ultimate limit for would be , a variant of the Kolmogorov complexity of , namely, the length of the shortest program (for the given hardware) that computes an output starting with [37], [45], [80], [110].”
BackgroundInfluentialA Novel Connectionist System for Unconstrained Handwriting Recognition
Graves, A., Liwicki, M., Fernandez, S., Bertolami, R., Bunke, H., & Schmidhuber, J. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.137 · Cited by 1,983
Self-citationAdvances in Neural Information Processing Systems 18
Murray, I., Mackay, D., Zoubin Ghahramani, A., & Skilling, J. (2006) · Cited by 1,927
Theory of Optimal Experiments.
Herzberg, A. M., Wynn, H. P., Fedorov, V. V., Studden, W. J., & Klimko, E. M. (1972) · Biometrika · doi:10.2307/2334826 · Cited by 1,897 · cited 2×
“, [15]? To optimize a function may require expensive data evaluations.”
BackgroundMethodAn Introduction to Kolmogorov Complexity and Its Applications
Li, M., & Vitányi, P. (1997) · Springer eBooks · doi:10.1007/978-1-4757-2606-0 · Cited by 1,775 · cited 9×
“The ultimate limit for would be , a variant of the Kolmogorov complexity of , namely, the length of the shortest program (for the given hardware) that computes an output starting with [37], [45], [80], [110].”
InfluentialA formal theory of inductive inference. Part II
Solomonoff, R. J. (1964) · Information and Control · doi:10.1016/s0019-9958(64)90131-7 · Cited by 1,758 · cited 4×
“Consider Solomonoff’s theoretically optimal, universal way of predicting the future [32], [45], [109], [110].”
BackgroundInfluentialSeung, H. S., Opper, M., & Sompolinsky, H. (1992) · Proceedings of the fifth annual workshop on Computational learning theory · doi:10.1145/130385.130417 · Cited by 1,651 · cited 1×
“, [2], [12], [15], [33], [47], [55], and [105], asking which data points to evaluate next to maximize information gain, typically (but not necessarily) using one”
BackgroundVent, W. (1975) · Feddes Repertorium · doi:10.1002/fedr.19750860506 · Cited by 1,331
Information-based objective functions for active data selection
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.4.590 · Cited by 1,221 · cited 1×
“, [2], [12], [15], [33], [47], [55], and [105], asking which data points to evaluate next to maximize information gain, typically (but not necessarily) using one”
BackgroundA formal theory of inductive inference. Part I
Solomonoff, R. J. (1964) · Information and Control · doi:10.1016/s0019-9958(64)90223-2 · Cited by 1,192
Cilibrasi, R., & Vitanyi, P. M. B. (2005) · IEEE Transactions on Information Theory · doi:10.1109/tit.2005.844059 · Cited by 1,188
Intrinsic Motivation Systems for Autonomous Mental Development
Oudeyer, P.-Y., Kaplan, F., & Hafner, V. V. (2007) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2006.890271 · Cited by 1,126
An Information Measure for Classification
Wallace, C. S., & Boulton, D. M. (1968) · The Computer Journal · doi:10.1093/comjnl/11.2.185 · Cited by 933
Raskin, V. (1979) · Annual Meeting of the Berkeley Linguistics Society · doi:10.3765/bls.v5i0.2164 · Cited by 912 · cited 1×
“, [58]) also focus on the element of surprise, they lack the essential concept of novel pattern detection measured by compression progress due to learning.”
BackgroundAdvances in Neural Information Processing Systems 7
Tesauro, G., Touretzky, D. S., & Leen, T. K. (1995) · Cited by 869
Action and behavior: a free-energy formulation
Friston, K. J., Daunizeau, J., Kilner, J., & Kiebel, S. J. (2010) · Biological Cybernetics · doi:10.1007/s00422-010-0364-z · Cited by 866 · cited 1×
“[19] (2010) also propose an approach which, in many ways, seems similar to ours, based on free energy minimization and predictive coding.”
MethodWhat is intrinsic motivation? A typology of computational approaches
Oudeyer, P.-Y. (2007) · Frontiers in Neurorobotics · doi:10.3389/neuro.12.006.2007 · Cited by 811 · cited 9×
“The authors of [52] originally intended to present 2a and 2b as examples of 2, not as sub-categories (P.”
BackgroundMethodInfluentialDayan, P., & Hinton, G. E. (1992) · UCL Discovery (University College London) · Cited by 769
The child's construction of reality
Piaget, J. (1954) · Medical Entomology and Zoology · Cited by 622
Intrinsically Motivated Reinforcement Learning
Singh, S., Barto, A. G., & Chentanez, N. (2005) · doi:10.21236/ada440280 · Cited by 610 · cited 4×
“[107] also used intrinsic rewards proportional to prediction errors as in Section III-A [71], employing a different type of reward maximizer based on the option framework which can be used to specify subgoals.”
BackgroundMethodInfluentialBirkhoff, G. D. (1933) · Harvard University Press eBooks · doi:10.4159/harvard.9780674734470 · Cited by 576
Curious model-building control systems
Schmidhuber, J. (1991) · [Proceedings] 1991 IEEE International Joint Conference on Neural Networks · doi:10.1109/ijcnn.1991.170605 · Cited by 541 · cited 10×
“But practical implementations so far have looked only at par s of the history for efficiency reasons: The systems described in Sections III, III-A, III-B, III-C, III-D [71], [70], [111], [79] used online settings (one prediction per time step, and cons tant computational effort per prediction), non-universal adap tive…”
BackgroundMethodInfluentialSelf-citationBayesian Surprise Attracts Human Attention
Itti, L., & Baldi, P. (2005) · neural information processing systems · Cited by 533
NOVELTY AND CURIOSITY AS DETERMINANTS OF EXPLORATORY BEHAVIOUR 1
BERLYNE, D. E. (1950) · British Journal of Psychology. General Section · doi:10.1111/j.2044-8295.1950.tb00262.x · Cited by 529 · cited 1×
“1) Developmental Psychology: In the 1950s Berlyne and other psychologists revisited the idea of curiosity as the motivation for exploratory behavior [5], [6], emphasizing the importance of novelty [5] and nonhomeostatic drives [25].”
BackgroundEstimation and Inference by Compact Coding
Wallace, C. S., & Freeman, P. R. (1987) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1987.tb01695.x · Cited by 523 · cited 3×
“[115], [116] comparing the number of bits”
BackgroundUniversal Artificial Intellegence
Hutter, M. (2005) · Texts in Theoretical Computer Science An EATCS Series · doi:10.1007/b138233 · Cited by 523 · cited 9×
“AIXI uses observations optimally [32]: the Bayes-optimal policy p based on the mixture ξ is selfoptimizing in the sense that its average utility value converges asymptotically for all μ ∈ M to the optimal value achieved by the Bayes-optimal policy p which knows μ in advance.”
BackgroundMethodInfluentialPfeifer, R., Soulié, F. F., Steels, L., & Schreter, Z. (1989) · Elsevier eBooks · Cited by 514
ZCS: A Zeroth Level Classifier System
Wilson, S. W. (1994) · Evolutionary Computation · doi:10.1162/evco.1994.2.1.1 · Cited by 505 · cited 1×
“Reward propagation procedures of traditional RL such as Q-learning [117] or RL economies & bucket brigade systems [30], [66], [65], [120] may be viewed as translating rare”
BackgroundIntrinsically Motivated Learning of Hierarchical Collections of Skills
Barto, A. G., Singh, S. P., & Chentanez, N. (2004) · Cited by 481
Multiple model-based reinforcement learning
Doya, K., Samejima, K., Katagiri, K.-I., & Kawato, M. (2002) · Neural Computation · doi:10.1162/089976602753712972 · Cited by 476 · cited 2×
“Nonetheless, an interesting aspect of IAC's greedy reward maximizer, is that it splits the state space into regions, reminiscent of algorithms by Doya [14] and Moore [49]; this might make learning more robust in certain situations. covers some of the topics discussed in the first paper on this subject: [71] (our Secti…”
Artificial General Intelligence
(2007) · Cognitive Technologies · doi:10.1007/978-3-540-68677-4 · Cited by 473
A Possibility for Implementing Curiosity and Boredom in Model-Building Neural Controllers
Schmidhuber, J. (1991) · From Animals to Animats · doi:10.7551/mitpress/3115.003.0030 · Cited by 421 · cited 29×
“To improve their models, they acquire skills to create / discover more data predictable or compressible in hitherto unknown ways [67], [71], [69], [70 ], [111], [77], [79], [85], [92], [93], [97], [96], [94], [99].”
BackgroundMethodInfluentialSelf-citationExploiting Open-Endedness to Solve Problems Through the Search for Novelty
Lehman, J., & Stanley, K. O. (2008) · Cited by 408
Learning Complex, Extended Sequences Using the Principle of History Compression
Schmidhuber, J. (1992) · Neural Computation · doi:10.1162/neco.1992.4.2.234 · Cited by 398 · cited 1×
“Given the predictor, only the wrongly predicted plus information about the corresponding time steps are necessary to reconstruct input history , e.g., [73].”
BackgroundSelf-citationDolitsky, M. (1986) · Journal of Pragmatics · doi:10.1016/0378-2166(86)90093-7 · Cited by 388
Complexity-based induction systems: Comparisons and convergence theorems
Solomonoff, R. (1978) · IEEE Transactions on Information Theory · doi:10.1109/tit.1978.1055913 · Cited by 388
Why am and eurisko appear to work
Lenat, D. B., & Brown, J. S. (1984) · Artificial Intelligence · doi:10.1016/0004-3702(84)90016-x · Cited by 307 · cited 1×
“Lenat’s discovery system EURISKO [41], [42] has a preprogrammed interestingness measure which was observed to become more an more inappropriate (“stagnation” problem) as EURISKO created new concepts from old ones with the help of human intervention.”
BackgroundEmpowerment: A Universal Agent-Centric Measure of Control
Klyubin, A. S., Polani, D., & Nehaniv, C. L. (2005) · 2005 IEEE Congress on Evolutionary Computation · doi:10.1109/cec.2005.1554676 · Cited by 274 · cited 2×
“’s seemingly related approach to intrinsic motivation [36] of 2005 tries to maximize empowerment by maximizing the information an agent could potentially “inject” into its future sensory inputs via a sequence of actions.”
BackgroundDevelopmental robotics, optimal artificial curiosity, creativity, music, and the fine arts
Schmidhuber, J. (2006) · Connection Science · doi:10.1080/09540090600768658 · Cited by 268
Self-citationMoore, A. W., & Atkeson, C. G. (1995) · Machine Learning · doi:10.1023/a:1022656217772 · Cited by 259 · cited 2×
“Nonetheless, an interesting aspect of IAC's greedy reward maximizer, is that it splits the state space into regions, reminiscent of algorithms by Doya [14] and Moore [49]; this might make learning more robust in certain situations. covers some of the topics discussed in the first paper on this subject: [71] (our Secti…”
MethodInformation Theory and Esthetic Perception
Wenk, A. B., Moles, A., & Cohen, J. E. (1968) · Notes · doi:10.2307/894007 · Cited by 182
Schmidhuber, J., Zhao, J., & Wiering, M. (1997) · Machine Learning · doi:10.1023/a:1007383707642 · Cited by 169
Self-citationBalcan, M.-F., Beygelzimer, A., & Langford, J. (2009) · Journal of Computer and System Sciences · doi:10.1016/j.jcss.2008.07.003 · Cited by 169
Advances in Evolutionary Computing: Theory and Applications
(2003) · Natural Computing Series · doi:10.1007/978-3-642-18965-4 · Cited by 167
Accelerated Neural Evolution through Cooperatively Coevolved Synapses
GomezFaustino, SchmidhuberJürgen, & MiikkulainenRisto (2008) · Journal of Machine Learning Research · doi:10.5555/1390681.1390712 · Cited by 164
Self-citationWiering, M., & Schmidhuber, J. (1997) · Adaptive Behavior · doi:10.1177/105971239700600202 · Cited by 163 · cited 3×
“apparently being unaware of previous work on hierarchical RL systems that can discover their own subgoals [60], [102], [61], [118], [79], [1].”
BackgroundMethodSelf-citationOptimal Ordered Problem Solver
Schmidhuber, J. (2004) · Machine Learning · doi:10.1023/b:mach.0000015880.99707.b2 · Cited by 162 · cited 3×
“A runtime-dependent performance measure inspired by concepts of optimal universal search [43], [81], [82], [85], [96], [99] is”
BackgroundMethodSelf-citationQuery-based learning applied to partially trained multilayer perceptrons
Hwang, J.-N., Choi, J. J., Oh, S., & Marks, R. J. (1991) · IEEE Transactions on Neural Networks · doi:10.1109/72.80299 · Cited by 152 · cited 1×
“, [15], [33], [47], [105], [55], [12], [2], asking which data points to evaluate next to maximize information gain, typically (but not necessari ly) using 1 step look-ahead, assuming all data point evaluation s are equally costly.”
BackgroundThe Speed Prior: A New Simplicity Measure Yielding Near-Optimal Computable Predictions
Schmidhuber, J. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45435-7_15 · Cited by 145 · cited 6×
“One may increase the theoretical power of the scheme by aug- menting by certain nonenumerable but limit-computable distributions [80], or restrict it such that it becomes computable, e.g., by assuming the world is computed by some unknown, but deterministic computer program sampled from the Speed Prior [81], which ass…”
BackgroundMethodInfluentialSelf-citationSchmidhuber, J. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-02565-5_4 · Cited by 139 · cited 10×
“They embody approximations of a simple, but general, formal theory of creativity and curiosity and inte restingness and fun, explaining essential aspects of human or no nhuman intelligence, including selective attention, scien e, art, music, humor [85], [92], [97], [96], [94].”
BackgroundMethodInfluentialSelf-citationEfficient Non-linear Control Through Neuroevolution
Gomez, F. J., Schmidhuber, J., & Miikkulainen, R. P. (2006) · Lecture notes in computer science · doi:10.1007/11871842_64 · Cited by 129 · cited 1×
“a) A traditional Q-learner [117] able to deal with delayed rewards as long as the environment is fully observable, like in Section III-B? A more limited 1-step look-ahead learner [52] that will break down in presence of delayed intrinsic rewards? A more sophisticated RL algorithm for delayed rewards in partially obser…”
MethodSelf-citationProperties of the Bucket Brigade
Holland, J. H. (1985) · international conference on Genetic algorithms · Cited by 127
Information Theory and Esthetic Perception
Kraehenbuehl, D., & ROHMER., M. A. A. Y. E. (1967) · Journal of Music Theory · doi:10.2307/842953 · Cited by 124
A computer scientist's view of life, the universe, and everything
Schmidhuber, J. (1997) · Lecture Notes in Computer Science · doi:10.1007/bfb0052088 · Cited by 119 · cited 1×
“The entire universe is regular an d apparently rather benign [74], [78], [88], [90]: every phot on behaves the same way; gravity is the same on Jupiter and Mars, mountains usually don’t move overnight but tend to remain where they are, etc.”
BackgroundSelf-citationHierarchical Reinforcement Learning Based on Subgoal Discovery and Subpolicy Specialization
Bakker, B., & Schmidhuber, J. (2004) · Cited by 111
Self-citationGödel Machines: Fully Self-referential Optimal Universal Self-improvers
Schmidhuber, J. (2007) · Cognitive Technologies · doi:10.1007/978-3-540-68677-4_7 · Cited by 110 · cited 2×
“To take the consumed computation time into account in a general, optimal way, one may use the recent Gödel machines [83], [86], [84], [98] instead.”
MethodSelf-citationIntrinsically Motivated Hierarchical Skill Learning in Structured Environments
Vigorito, C. M., & Barto, A. G. (2010) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2010.2050205 · Cited by 105
Neural Network Exploration Using Optional Experiment Design,
Cohn, D. (1994) · doi:10.21236/ada290058 · Cited by 99
Theory formation by heuristic search
Lenat, D. B. (1983) · Artificial Intelligence · doi:10.1016/s0004-3702(83)80004-6 · Cited by 98 · cited 2×
“Lenat's discovery system EURISKO [41], [42] has a preprogrammed interestingness measure which was observed to become more an more inappropriate ( " stagnation " problem) as EURISKO created new concepts from old ones with the help of human intervention.”
BackgroundSCHMIDHUBER, J. (2002) · International Journal of Foundations of Computer Science · doi:10.1142/s0129054102001291 · Cited by 98 · cited 3×
“The ultimate limit for would be , a variant of the Kolmogorov complexity of , namely, the length of the shortest program (for the given hardware) that computes an output starting with [37], [45], [80], [110].”
BackgroundSelf-citationLearning from Logged Implicit Exploration Data
Strehl, A. L., Langford, J. C., Kakade, S. M., & Li, L. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1003.0120 · Cited by 97 · cited 1×
“[112], our more general systems measure and maximize algorithmic [37], [45], [80], [110] novelty (learnable, but previously unknown compressibility or predictability) of self-generated spatio–temporal patterns in the history of data and actions [85], [92], [96], [97].”
BackgroundBarto, A. G., Lewis, R. L., & Singh, S. P. (2009) · eScholarship (California Digital Library) · Cited by 96
Sustaining diversity using behavioral information distance
Gomez, F. J. (2009) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/1569901.1569918 · Cited by 94 · cited 2×
“Recent related work in the field of evolutionary computation aims at increasing diversity within populations of individ uals [40], [21], [23].”
BackgroundMethodFrom unknown sensors and actuators to actions grounded in sensorimotor perceptions
Olsson, L., Nehaniv, C. L., & Polani, D. (2006) · Connection Science · doi:10.1080/09540090600768542 · Cited by 93 · cited 1×
“Other researchers in the nascent field of developmental robotics [9], [57], [38], [113], [64], [20], [51], [52], [27 ], [26] and intrinsic reward also took up the basic idea, for example, Oudeyer et al [53].”
BackgroundSequential neural text compression
Schmidhuber, J., & Heil, S. (1996) · IEEE Transactions on Neural Networks · doi:10.1109/72.478398 · Cited by 93 · cited 3×
“3) Let some (possibly application-dependent) compressor improvement algorithm (such as a learning algorithm for an adaptive neural network predictor, possibly triggered by a controller action) use to obtain a hopefully better compressor (such as a neural net with the same size and the same constant computational effor…”
BackgroundSelf-citationAn on-line algorithm for dynamic reinforcement learning and planning in reactive environments
Schmidhuber, J. (1990) · 1990 IJCNN International Joint Conference on Neural Networks · doi:10.1109/ijcnn.1990.137723 · Cited by 89 · cited 1×
“1990: Non-traditional RL (without restrictive Markovian assumptions [72]) based on an adaptive recurrent neural network as a predictive world model [68] is used to maximize the controller’s intrinsic reward, which is proportio nal to the model’s prediction errors [67], [71].”
MethodSelf-citationReinforcement Learning in Markovian and Non-Markovian Environments
Schmidhuber, J. (1990) · Cited by 89
Self-citationA Local Learning Algorithm for Dynamic Feedforward and Recurrent Networks
SCHMIDHUBER, J. (1989) · Connection Science · doi:10.1080/09540098908915650 · Cited by 87 · cited 1×
“[30], [65], [66], [120] may be viewed as translating rare ex-”
BackgroundSelf-citationSchmidhuber, J. (2009) · Cognitive Computation · doi:10.1007/s12559-009-9014-y · Cited by 87 · cited 10×
“The corresponding algorithmic framework uses general RL (Section II-G, [32], and [98]) to maximize not only external reward for achieving goals such as the satisfaction of hunger and thirst, but also intrinsic reward for learning a better world model, by creating/discovering/learning novel patterns in the growing hist…”
BackgroundMethodInfluentialSelf-citationWiering, M., & Schmidhuber, J. (1998) · Machine Learning · doi:10.1023/a:1007562800292 · Cited by 83 · cited 1×
“Others postpone updates until needed [119].”
Self-citationSchmidhuber, J. (1997) · Leonardo · doi:10.2307/1576418 · Cited by 80 · cited 2×
“How does the theory informally explain the motivation to create or perceive art and music [75], [76], [85], [92], [94], [96], [97]? For example, why are some melodies more interesting or aesthetically rewarding than others? Not the one the listener (composer) just heard (played) twenty times in a row.”
BackgroundSelf-citationAlgorithmic Theories of Everything
Schmidhuber, J. (2000) · arXiv (Cornell University) · doi:10.48550/arxiv.quant-ph/0011122 · Cited by 78 · cited 1×
“The entire universe is regular an d apparently rather benign [74], [78], [88], [90]: every phot on behaves the same way; gravity is the same on Jupiter and Mars, mountains usually don’t move overnight but tend to remain where they are, etc.”
BackgroundSelf-citationKnows what it knows: a framework for self-aware learning
Li, L., Littman, M. L., & Walsh, T. J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390228 · Cited by 78
Evolving internal reinforcers for an intrinsically motivated reinforcement-learning robot
Schembri, M., Mirolli, M., & Baldassarre, G. (2007) · 2007 IEEE 6th International Conference on Development and Learning · doi:10.1109/devlrn.2007.4354052 · Cited by 77 · cited 1×
“They speed up skill learning by rewarding a top level, module-selecting RL agent in proport ion to the TD error of the selected module [63]—compare Section III-B.”
BackgroundEinführung in die informationstheoretische Ästhetik: Grundlegung und Anwendung in der Texttheorie
Bense, M. (1998) · J.B. Metzler eBooks · doi:10.1007/978-3-476-03716-9_3 · Cited by 69 · cited 2×
“2) Aesthetics Theory: The closely related field of aesthetics theory [7], [48], [4], [16], [50], [18] emerged even earlier in the 1930s.”
BackgroundMethodSchmidhuber, J. (2007) · Springer eBooks · doi:10.1007/978-3-540-75488-6_3 · Cited by 66 · cited 10×
“They embody approximations of a simple, but general, formal theory of creativity and curiosity and inte restingness and fun, explaining essential aspects of human or no nhuman intelligence, including selective attention, scien e, art, music, humor [85], [92], [97], [96], [94].”
BackgroundMethodInfluentialSelf-citationSchmidhuber, J. (2003) · Natural Computing Series · doi:10.1007/978-3-642-18965-4_23 · Cited by 65 · cited 22×
“Note that some actions may actually trigger learning algorithms that compute changes of the compressor and the controller’s policy, such as in Section III-D [79].”
BackgroundInfluentialSelf-citationSchmidhuber, J. (2009) · Cited by 62
Self-citationBootstrap learning of foundational representations
Kuipers, B. J., Beeson, P., Modayil, J., & Provost, J. (2006) · Connection Science · doi:10.1080/09540090600768484 · Cited by 62 · cited 1×
“Other researchers in the nascent field of developmental robotics [9], [57], [38], [113], [64], [20], [51], [52], [27 ], [26] and intrinsic reward also took up the basic idea, for example, Oudeyer et al [53].”
BackgroundÄsthetik als Informationsverarbeitung
Nake, F. (1974) · Springer eBooks · doi:10.1007/978-3-7091-7097-7 · Cited by 49 · cited 2×
“2) Aesthetics Theory: The closely related field of aesthetics theory [4], [7], [16], [18], [48], [50] emerged even earlier in the 1930s.”
BackgroundSchmidhuber, J. (1997) · SUPSI ARIS · doi:10.71910/supsi.7455 · Cited by 39
Self-citationIntrinsically motivated hierarchical manipulation
Hart, S., Sen, S., & Grupen, R. A. (2008) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2008.4543796 · Cited by 35 · cited 1×
“Other researchers in the nascent field of developmental robotics [9], [20], [26], [27], [38], [51], [52], [57], [64], [113] and intrinsic reward also took up the basic idea, for example, Oudeyer et al.”
Background2006: Celebrating 75 Years of AI - History and Outlook: The Next 25 Years
Schmidhuber, J. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-77296-5_4 · Cited by 34 · cited 1×
“Unlike the previous nonself-referential methods based on hardwired proof searche rs [32], Gödel machines not only boast an optimal order of complexity but can optimally reduce (through self-changes ) any slowdowns hidden by theO()-notation, provided the utility of such speed-ups is provable [87], [91], [89].”
MethodSelf-citationNew Millennium AI and the Convergence of History
Schmidhuber, J. (2007) · Studies in computational intelligence · doi:10.1007/978-3-540-71984-7_2 · Cited by 34 · cited 1×
“Unlike the previous nonself-referential methods based on hardwired proof searche rs [32], Gödel machines not only boast an optimal order of complexity but can optimally reduce (through self-changes ) any slowdowns hidden by theO()-notation, provided the utility of such speed-ups is provable [87], [91], [89].”
MethodSelf-citationCompletely Self-referential Optimal Reinforcement Learners
Schmidhuber, J. (2005) · Lecture notes in computer science · doi:10.1007/11550907_36 · Cited by 33 · cited 3×
“To take the consumed computation time into account in a general, optimal way, one may use the recent Gödel machine s [83], [84], [86], [98] instead.”
BackgroundMethodSelf-citationThe New AI: General & Sound & Relevant for Physics
Schmidhuber, J. (2007) · Cognitive technologies · doi:10.1007/978-3-540-68677-4_6 · Cited by 33 · cited 1×
“Unlike the previous nonself-referential methods based on hardwired proof searche rs [32], Gödel machines not only boast an optimal order of complexity but can optimally reduce (through self-changes ) any slowdowns hidden by theO()-notation, provided the utility of such speed-ups is provable [87], [91], [89].”
MethodSelf-citationTowards autonomous sensor and actuator model induction on a mobile robot
Stronger, D., & Stone, P. (2006) · Connection Science · doi:10.1080/09540090600768690 · Cited by 30 · cited 1×
“Other researchers in the nascent field of developmental robotics [9], [57], [38], [113], [64], [20], [51], [52], [27 ], [26] and intrinsic reward also took up the basic idea, for example, Oudeyer et al [53].”
BackgroundDeveloping navigation behavior through self-organizing distinctive-state abstraction
Provost, J., Kuipers, B. J., & Miikkulainen, R. P. (2006) · Connection Science · doi:10.1080/09540090600768609 · Cited by 29 · cited 1×
“Other researchers in the nascent field of developmental robotics [9], [20], [26], [27], [38], [51], [52], [57], [64], [113] and intrinsic reward also took up the basic idea, for example, Oudeyer et al.”
BackgroundArtificial General Intelligence
Schmidhuber, J., Looks, M., & Þórisson, K. R. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22887-2 · Cited by 28
Self-citationIntroduction to developmental robotics
Meeden, L., & Blank, D. (2006) · Connection Science · doi:10.1080/09540090600806631 · Cited by 27
Learning Mackey-Glass from 25 examples, Plus or Minus 2
Plutowski, M., Cottrell, G. W., & White, H. (1993) · neural information processing systems · Cited by 27
Ring, M. B. (1991) · Elsevier eBooks · doi:10.1016/b978-1-55860-200-7.50071-4 · Cited by 26
The Development of Hierarchical Knowledge in Robot Systems
Hart, S. W. (2009) · Scholarworks (University of Massachusetts Amherst) · doi:10.7275/782w-wa28 · Cited by 25 · cited 1×
“Other researchers in the nascent field of developmental robotics [9], [20], [26], [27], [38], [51], [52], [57], [64], [113] and intrinsic reward also took up the basic idea, for example, Oudeyer et al.”
BackgroundLearning acceptable windows of contingency
Gold, K., & Scassellati, B. M. (2006) · Connection Science · doi:10.1080/09540090600768435 · Cited by 25 · cited 1×
“Other researchers in the nascent field of developmental robotics [9], [20], [26], [27], [38], [51], [52], [57], [64], [113] and intrinsic reward also took up the basic idea, for example, Oudeyer et al.”
BackgroundPlanning Simple Trajectories Using Neural Subgoal Generators
Schmidhuber, J., & Wahnsiedler, R. (1993) · From Animals to Animats 2 · doi:10.7551/mitpress/3116.003.0027 · Cited by 24 · cited 3×
“apparently being unaware of previous work on hierarchical R L systems that can discover their own subgoals [60], [102], [6 1], [118], [79], [1].”
BackgroundMethodSelf-citationKybernetische Analysen : subjektiver Sachverhalte
Frank, H. (1964) · Medical Entomology and Zoology · Cited by 23
Adaptation in Constant Utility Non-Stationary Environments.
Littman, M. L., & Ackley, D. H. (1991) · ICGA · Cited by 22
Gödel Machines: Towards a Technical Justification of Consciousness
Schmidhuber, J. (2005) · Lecture notes in computer science · doi:10.1007/978-3-540-32274-0_1 · Cited by 19 · cited 2×
“To take the consumed computation time into account in a general, optimal way, one may use the recent Gödel machines [83], [86], [84], [98] instead.”
MethodSelf-citationDriven by Compression Progress
Schmidhuber, J. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-85563-7_2 · Cited by 14 · cited 1×
“To improve their models, they acquire skills to create / discover more data predictable or compressible in hitherto unknown ways [67], [71], [69], [70 ], [111], [77], [79], [85], [92], [93], [97], [96], [94], [99].”
BackgroundSelf-citationArtificial Scientists a Artists Based on the Formal Theory of Creativity
Schmidhuber, J. (2010) · Artificial General Intelligence · doi:10.2991/agi.2010.32 · Cited by 13 · cited 5×
“To improve their models, they acquire skills to create / discover more data predictable or compressible in hitherto unknown ways [67], [71], [69], [70 ], [111], [77], [79], [85], [92], [93], [97], [96], [94], [99].”
BackgroundMethodInfluentialSelf-citationMeasuring and Optimizing Behavioral Complexity for Evolutionary Reinforcement Learning
Gomez, F. J., Togelius, J., & Schmidhuber, J. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04277-5_77 · Cited by 12
Self-citationDon't forget randomness is still just a hypothesis
Schmidhuber, J. (2006) · Nature · doi:10.1038/439392d · Cited by 12
Self-citationDecomposing infants’ object representations: A dual-route processing account
Schlesinger, M. (2006) · Connection Science · doi:10.1080/09540090600768740 · Cited by 11 · cited 1×
“Other researchers in the nascent field of developmental robotics [9], [57], [38], [113], [64], [20], [51], [52], [27 ], [26] and intrinsic reward also took up the basic idea, for example, Oudeyer et al [53].”
BackgroundSchmidhuber, J. (1989) · mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) · Cited by 9
Self-citationSchmidhuber, J. (2009) · Jidō seigyo/Jido seigyo · Cited by 9
Self-citationA Possibility for Implementing Curiosity and Boredom in Model-Building Neural Controllers
Meyer, J.-A., & Wilson, S. W. (1991) · Cited by 9
The classification of spikes in EEG recordings using features derived from ICA
Marzia De Lucia, E., Fritschy, J., Dayan, P., & Holder, D. S. (2006) · International Conference on Advances in Medical, Signal and Information Processing (MEDSIP) · doi:10.1049/cp:20060397 · Cited by 6
Strong Asymptotic Assertions for Discrete MDL in Regression and ClassificationPreprint
Poland, J., & Hütter, M. (2005) · arXiv (Cornell University) · doi:10.48550/arxiv.math/0502315 · Cited by 6
Research works printed but not linked: 24
- kullback, 1959, Statistics and Information Theory
- hutter, 2004, Universal Artificial Intelligence Sequential Decisions Based on Algorithmic Probability
- holland, 1975, Adaptation in Natural and Artificial Systems
- schmidhuber, 2010, How to Learn Programs with Artificial Recurrent Neural Networks
- harlow, 1950, novelty and curiosity as determinants of exploratory behavior, J Exp Psychol, 41, 68
- berlyne, 1950, novelty and curiosity as determinants of exploratory behavior, Brit J Psychol, 41, 68
- 10.1007/bf00993591 · doi:10.1007/bf00993591 · Paper page
- levin, 1973, universal sequential search problems, Problems Inform Trans, 9, 265
- schmidhuber, 1991, Advances in Neural Information Processing Systems 3, nips 3, 500
- schmidhuber, 1997, Femmes Fractales
- schmidhuber, 2002, Advances in Evolutionary Computing, 579
- ring, 1994, Continual Learning in Reinforcement Environments
- schmidhuber, 1990, Dynamische neuronale Netze und das fundamentale raumzeitliche Lernproblem
- schmidhuber, 1991, Adaptive Curiosity and Adaptive Confidence
- schmidhuber, 2009, Compression Progress The Algorithmic Principle Behind Curiosity and Creativity (With Applications of the Theory of Humor) 40 min Video of Invited Talk at Singularity Summit 2009
- schmidhuber, 2009, Multiple Ways to Design Research Research Cases that Reshape the Design Discipline, 98
- singh, 2005, Advances in neural information processing systems, 17
- schwefel, 1974, Numerische Optimierung von Computer-Modellen
- schmidhuber, 2007, alle berechenbaren universen (all computable universes), Spektrum der Wissenschaft, 75
- storck, 1995, reinforcement driven information acquisition in non-deterministic environments, Proc Int Conf Artif Neural Netw, 2, 159
- brafman, 2002, R-MAX—A general polynomial time algorithm for near-optimal reinforcement learning, J Mach Learn Res, 3, 213
- frank, 2002, sthetische Information Estetika informacio Eine Einfhrung in die Kybernetische sthetik
- franke, 1979, Kybernetische sthetik Phnomen Kunst
- schmidhuber, 2006, Artificial General Intelligence, 175
Cited by
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815
“Note that ICM-pixels is representative of previous methods which compute information gain by directly using the observation space (Schmidhuber, 2010; Stadie et al., 2015).”
BackgroundMethodInfluentialAn Introduction to Deep Reinforcement Learning
Vincent, F.-L., Peter, H., Riashat, I., Marc, G. B., & Joelle, P. (2018) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000071 · Cited by 1,286
“Another common approach is to have a directed scheme thanks to exploration rewards given to the agent via heuristics that estimate novelty (Schmidhuber, 2010; Stadie et al., 2015; Houthooft et al., 2016).”
REBUS and the anarchic brain: Toward a unified model of the brain action of psychedelics
Carhart-Harris, R. L., & Friston, K. J. (2019) · Pharmacological Reviews · doi:10.1124/pr.118.017160 · Cited by 1,180
“, 2017) and is consistent with an explorative search strategy, which entails exploring novelty for the sake of significant knowledge gain (also known as intrinsic motivation in robotics) (Oudeyer and Kaplan, 2007; Schmidhuber, 2010; Barto et al., 2013).”
BackgroundMulti-agent deep reinforcement learning: a survey
Gronauer, S., & Diepold, K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-09996-w · Cited by 957
“One way to realize this is with intrinsic motivation, which is based on the concept of maximizing an internal reinforcement signal by actively discovering novel or surprising patterns (Chentanez et al. 2005; Oudeyer and Kaplan 2007; Schmidhuber 2010).”
BackgroundHierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Kulkarni, T. D., Narasimhan, K., Saeedi, A., & Tenenbaum, J. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06057 · Cited by 770
Metadata
- Authors
- Schmidhuber, J.
- Affiliations
- Dalle Molle Institute for Artificial Intelligence Research (CH)
- Corresponding author
- Schmidhuber, Jürgen
- ORCID
- Schmidhuber, Jürgen
- Venue
- IEEE Transactions on Autonomous Mental Development
- Published
- 2010-09
- Volume
- 2
- Issue
- 3
- Pages
- 230-247
- ISSN
- 1943-0604 (ISSN-L)
- 1943-0612
- OpenAlex
- W2034806191
- Cited by
- 747
- References
- 126
- Citations by year
- 2026: 29 · 2025: 64 · 2024: 46 · 2023: 50 · 2022: 60 · 2021: 69 · 2020: 92 · 2019: 72 · 2018: 55 · 2017: 37 · 2016: 26 · 2015: 24 · 2014: 30 · 2013: 28 · 2012: 41
- FWCI
- 31.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Autonomous Mental Development (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Creativity in Education and Neuroscience
- Keywords
- intrinsic motivation, creativity, fun, intrinsic rewards, active learning (machine learning), novelty seeking, surprise, pattern discovery, data compression, online prediction, autonomous agents, robots, aesthetics, humor, intelligence, limited computational resources, continuous-time formulation, discrete-time formulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.