Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010)
The DOI matches its Crossref record; 125 of the 126 references it cites were found in a registry or an index.
Abstract
The simple, but general formal theory of fun and intrinsic motivation and creativity (1990-2010) is based on the concept of maximizing intrinsic reward for the active creation or discovery of novel, surprising patterns allowing for improved prediction or data compression. It generalizes the traditional field of active learning, and is related to old, but less formal ideas in aesthetics theory and developmental psychology. It has been argued that the theory explains many essential aspects of intelligence including autonomous development, science, art, music, and humor. This overview first describes theoretically optimal (but not necessarily practical) ways of implementing the basic computational principles on exploratory, intrinsically motivated agents or robots, encouraging them to provoke event sequences exhibiting previously unknown, but learnable algorithmic regularities. Emphasis is put on the importance of limited computational resources for online prediction and compression. Discrete and continuous time formulations are given. Previous practical, but nonoptimal implementations (1991, 1995, and 1997-2002) are reviewed, as well as several recent variants by others (2005-2010). A simplified typology addresses current confusion concerning the precise nature of intrinsic motivation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This underwrites Bayesian experimental design [134] and active learning in statistics [135], intrinsic motivation and artificial curiosity in machine learning and robotics [136, 137, 138, 139, 140].”
From The free energy principle made simpler but not too simple (2023) · cites this paper as [137]
In the citing paper
“This suggests that risk-averse policies minimise expected complexity or computational cost [137].”
From The free energy principle made simpler but not too simple (2023) · cites this paper as [137]
In the citing paper
“Ideally, the learner learns to choose actions that will lead to the largest improvements in its prediction or data compression performance (Schmidhuber, 2010).”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Schmidhuber, 2010
In the citing paper
“In Schmidhuber (2010), this is done in the framework of reinforcement learning, and incorporates a mechanisms for the system to measure its own rate of learning.”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Schmidhuber, 2010
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 7×
“Another potential drawback is the nature of the particular RNN-based RL method.”
BackgroundInfluentialSelf-citationA Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
Holland, J. H. (1992) · The MIT Press eBooks · doi:10.7551/mitpress/1090.001.0001 · Cited by 47,002 · cited 1×
“iii) An evolutionary algorithm [22], [29], [59],”
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 7×
“Some Q-learning accelerators simply update pairs of actions and states with currently quickly changing Q-values more frequently than others (that is, Q-values with high first derivatives are favored).”
BackgroundMethodInfluentialReinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Neural networks for pattern recognition
(1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,538
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972 · cited 4×
“, some typically suboptimal RL method [35].”
BackgroundMethodInfluential
Research works printed but not linked: 24
- kullback, 1959, Statistics and Information Theory
- hutter, 2004, Universal Artificial Intelligence Sequential Decisions Based on Algorithmic Probability
- holland, 1975, Adaptation in Natural and Artificial Systems
- schmidhuber, 2010, How to Learn Programs with Artificial Recurrent Neural Networks
- harlow, 1950, novelty and curiosity as determinants of exploratory behavior, J Exp Psychol, 41, 68
- berlyne, 1950, novelty and curiosity as determinants of exploratory behavior, Brit J Psychol, 41, 68
- 10.1007/bf00993591 · doi:10.1007/bf00993591 · Paper page
- levin, 1973, universal sequential search problems, Problems Inform Trans, 9, 265
- schmidhuber, 1991, Advances in Neural Information Processing Systems 3, nips 3, 500
- schmidhuber, 1997, Femmes Fractales
- schmidhuber, 2002, Advances in Evolutionary Computing, 579
- ring, 1994, Continual Learning in Reinforcement Environments
- schmidhuber, 1990, Dynamische neuronale Netze und das fundamentale raumzeitliche Lernproblem
- schmidhuber, 1991, Adaptive Curiosity and Adaptive Confidence
- schmidhuber, 2009, Compression Progress The Algorithmic Principle Behind Curiosity and Creativity (With Applications of the Theory of Humor) 40 min Video of Invited Talk at Singularity Summit 2009
- schmidhuber, 2009, Multiple Ways to Design Research Research Cases that Reshape the Design Discipline, 98
- singh, 2005, Advances in neural information processing systems, 17
- schwefel, 1974, Numerische Optimierung von Computer-Modellen
- schmidhuber, 2007, alle berechenbaren universen (all computable universes), Spektrum der Wissenschaft, 75
- storck, 1995, reinforcement driven information acquisition in non-deterministic environments, Proc Int Conf Artif Neural Netw, 2, 159
- brafman, 2002, R-MAX—A general polynomial time algorithm for near-optimal reinforcement learning, J Mach Learn Res, 3, 213
- frank, 2002, sthetische Information Estetika informacio Eine Einfhrung in die Kybernetische sthetik
- franke, 1979, Kybernetische sthetik Phnomen Kunst
- schmidhuber, 2006, Artificial General Intelligence, 175
Cited by
Items 101–200 of 886
Boredom begets creativity: A solution to the exploitation–exploration trade-off in predictive coding
Gómez‐Ramírez, J., & Costa, T. (2017) · Biosystems · doi:10.1016/j.biosystems.2017.04.006 · Cited by 82
In search of boredom: beyond a functional account
Danckert, J. A., & Elpidorou, A. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.02.002 · Cited by 85
Model Based Reinforcement Learning for Atari
Kaiser, Ł., Babaeizadeh, M., Miłoś, P., Osiński, B., Campbell, R. H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., et al. (2020) · International Conference on Learning Representations · Cited by 335
Random curiosity-driven exploration in deep reinforcement learning
Li, J. K., Shi, X., Li, J., Zhang, X., & Wang, J. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.08.024 · Cited by 81
Intrinsic motivations and open-ended development in animals, humans, and robots: an overview
Baldassarre, G., Stafford, T., Mirolli, M., Redgrave, P., Ryan, R. M., & Barto, A. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00985 · Cited by 73
Information generation as a functional basis of consciousness
Kanai, R., Chang, A., Yu, Y., Magrans de Abril, I., Biehl, M., & Guttenberg, N. (2019) · Neuroscience of Consciousness · doi:10.1093/nc/niz016 · Cited by 72
“For example, an agent can compute in a model-based manner what would be the expected information gain (Schmidhuber 1991, 2010) or other forms of intrinsic motivation (Oudeyer 2007) in response to future (and therefore counterfactual) actions.”
BackgroundObject Learning Through Active Exploration
Ivaldi, S., Nguyen, S. M., Lyubova, N., Droniou, A., Padois, V., Filliat, D., Oudeyer, P., & Sigaud, O. (2014) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2013.2280614 · Cited by 79
“In robotics, they inspired the creation of meta-exploration mechanisms monitoring the evolution of learning performances [14], [27], [45], with heuristics defining the notion of interest used in an active learning framework [46], [47], [48].”
BackgroundA taxonomy of surprise definitions
Modirshanechi, A., Brea, J., & Gerstner, W. (2022) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2022.102712 · Cited by 84
“(30) The first definition is more consistent with the original definition of the Bayesian surprise [Baldi, 2002, Baldi and Itti, 2010, Schmidhuber, 2010] applied to our generative model because the belief before the observation should include the knowledge that the environment is volatile.”
BackgroundMethodInfluentialMind the gap: an attempt to bridge computational and neuroscientific approaches to study creativity
Wiggins, G. A., & Bhattacharya, J. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00540 · Cited by 82
“Finally, a significant series of work from the past decade and a half, which remains somewhat unfamiliar to psychologists and neuroscientists working on creativity, is that of Jürgen Schmidhuber (2010), whose theory relates a notion of intrinsic reward, derived from a rate of learning, to the discovery of new patterns…”
Intrinsically motivated reinforcement learning for human–robot interaction in the real-world
Qureshi, A. H., Nakamura, Y., Yoshikawa, Y., & Ishiguro, H. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.03.014 · Cited by 78
Shaping Belief States with Generative Environment Models for RL
Gregor, K., Rezende, D. J., Besse, F., Wu, Y. X., Merzić, H., & van den Oord, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.09237 · Cited by 50
“The idea of learning general world models to support decision-making is probably one of the most pervasive ideas in AI research, [30, 11, 41, 42, 14, 43, 44, 45, 46, 33, 30, 47, 48, 3, 49, 50, 51, 52].”
BackgroundBalancing new against old information: the role of puzzlement surprise in learning
Faraji, M., Preuschoff, K., & Gerstner, W. (2018) · Neural Computation · doi:10.1162/neco_a_01025 · Cited by 68
“Similar to earlier theories (Schmidhuber, 2010; Friston et al., 2015, 2017) we study learning rules that minimize the surprise if the same data point appears a second time, but in contrast to these earlier theories, we start from our definition of confidence-corrected surprise as opposed to information gain measures (…”
BackgroundMethodResultInfluentialSocially guided intrinsic motivation for robot learning of motor skills
Nguyen, S. M., & Oudeyer, P.-Y. (2014) · Autonomous Robots · doi:10.1007/s10514-013-9339-y · Cited by 46
“…Baldassarre 2011): parameters of motor policies are chosen for experimentation so that the observed consequences in the task space provide maximal improvement of the quality of the learned forward model, which is then inverted for control when needed (Oudeyer et al. 2007; Schmidhuber 2010, 1991).”
BackgroundStupacher, J., Matthews, T. E., Pando‐Naude, V., Elst, O. F. V., & Vuust, P. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.906190 · Cited by 83
“…this context, learning is thought to be inherently rewarding as it serves the innate drive for competence (Ryan and Deci, 2000) and satisfies the intrinsic motivation for information gain (i.e., curiosity; Schmidhuber, 2010; Kidd and Hayden, 2015; Friston et al., 2017; Gottlieb and Oudeyer, 2018).”
BackgroundChanging the Environment Based on Empowerment as Intrinsic Motivation
Salge, C., Glackin, C., & Polani, D. (2014) · Entropy · doi:10.3390/e16052789 · Cited by 81
“An early example is Schmidhuber’s work on Artificial curiosity [13,14] where an”
BackgroundLe, T. P., Vien, N. A., & Chung, T. (2018) · IEEE Access · doi:10.1109/access.2018.2854283 · Cited by 75
Intrinsically Motivated Learning Systems: An Overview
Baldassarre, G., & Mirolli, M. (2012) · Springer eBooks · doi:10.1007/978-3-642-32375-1_1 · Cited by 81
Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
Colas, C., Karch, T., Lair, N., Dussoux, J.-M., Moulin-Frier, C., Dominey, P. F., & Oudeyer, P.-Y. (2020) · arXiv · doi:10.48550/arxiv.2002.09253 · Cited by 64
“Algorithmic models of intrinsic motivation were successfully used in developmental robotics [55, 6], in reinforcement learning [16, 63] and more recently in deep RL [8, 56].”
MethodIntrinsically Motivated Learning of Real-World Sensorimotor Skills with Developmental Constraints
Oudeyer, P., Baranès, A., & Kaplan, F. (2012) · Springer eBooks · doi:10.1007/978-3-642-32375-1_13 · Cited by 86
Schmidhuber, J. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.04.008 · Cited by 88
“This insight led to lots of follow-up work [77].”
BackgroundResultInfluentialSelf-citationEducational Robotics and Robot Creativity: An Interdisciplinary Dialogue
Gubenko, A., Kirsch, C., Smilek, J. N., Lubart, T., & Houssemand, C. (2021) · Frontiers in Robotics and AI · doi:10.3389/frobt.2021.662030 · Cited by 70
“According to Schmidhuber (2010), the discovery of these novel regularities in curiosity-driven exploration would be marked by an impressive reduction in computational resources.”
BackgroundLearning Formal Mathematics From Intrinsic Motivation
Poesia, G., Broman, D., Haber, N., & Goodman, N. D. (2024) · neural information processing systems · doi:10.52202/079017-1362 · Cited by 63
Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement LearningPreprint
Iqbal, S., & Sha, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.12127 · Cited by 74
“Alternatively, some work has focused on defining intrinsic rewards for exploration based on inspiration from psychology (Oudeyer & Kaplan, 2009; Schmidhuber, 2010).”
Wen, W., & Haggard, P. (2019) · Cognition · doi:10.1016/j.cognition.2019.104074 · Cited by 69
“Further, neurorobotics approaches emphasise the importance of exploratory movements (Barto, Mirolli, & Baldassarre, 2013; Oudeyer & Kaplan, 2007; Parr & Friston, 2017; Schmidhuber, 2010).”
BackgroundModular active curiosity-driven discovery of tool use
Forestier, S., & Oudeyer, P.-Y. (2016) · 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759584 · Cited by 58
Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design
Dennis, M., Jaques, N., Vinitsky, E., Bayen, A., Russell, S., Critch, A., & Levine, S. (2020) · arXiv · doi:10.48550/arxiv.2012.02096 · Cited by 60
“One approach to produce policies which can transfer between environments is unsupervised RL (e.g., [34, 36]).”
BackgroundEffective Diversity in Population-Based Reinforcement Learning
Parker-Holder, J., Pacchiano, A., Choromanski, K., & Roberts, S. (2020) · arXiv · doi:10.48550/arxiv.2002.00632 · Cited by 60
“We believe this adaptive mechanism could also be used for count-based exploration methods or intrinsic rewards (Schmidhuber, 2010), and note very recent work using a similar approach to vary exploration in off-policy methods (Schaul et al., 2019; Badia et al., 2020) and model-based RL (Ball et al.,…”
MethodDeep Reinforcement Learning: A State-of-the-Art Walkthrough
Lazaridis, A., Fachantidis, A., & Vlahavas, I. P. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12412 · Cited by 78
“A well-known basis for exploration methods is curiosity-driven exploration, which arises from the concepts of curiosity (Silvia, 2012) and intrinsic motivation (Oudeyer & Kaplan, 2009; Schmidhuber, 2010).”
BackgroundBND*-DDQN: Learn to Steer Autonomously Through Deep Reinforcement Learning
Wu, K., Wang, H., Esfahani, M. A., & Yuan, S. (2019) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2019.2928820 · Cited by 65
“Typically, intrinsic rewards are used to quantify the intrinsic motivation [49] and the rationale is to predict the consequences resulted from taking an action [50], [51].”
BackgroundOkada, A., & Sheehy, K. (2020) · Frontiers in Education · doi:10.3389/feduc.2020.584351 · Cited by 65
“Fun as an inner joy of wellbeing and engagement is an important component to propitiate learning with the creation of new patterns that are interesting, surprising and meaningful (Schmidhuber, 2010) to involve students with formal education during uncertain time of post-pandemic.”
BackgroundOpen-Ended Learning Leads to Generally Capable Agents
Open Ended Learning Team, Stooke, A., Mahajan, A., Barros, C., Deck, C., Bauer, J., Sygnowski, J., Trebacz, M., Jaderberg, M., Mathieu, M., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.12808 · Cited by 56
“A number of methods attempt to use learning progress (Kaplan and Oudeyer, 2007; Schmidhuber, 2010) on a task as a way to decide whether the task should be trained on or not (Graves et al.”
MethodBaldassarre, G., Mannella, F., Fiore, V. G., Redgrave, P., GURNEY, K. N., & Mirolli, M. (2012) · Neural Networks · doi:10.1016/j.neunet.2012.09.015 · Cited by 61
Kompella, V. R., Luciw, M., & Schmidhuber, J. (2012) · Neural Computation · doi:10.1162/neco_a_00344 · Cited by 56
Self-citationActive inference models do not contradict folk psychology
Smith, R. S., Ramstead, M. J. D., & Kiefer, A. B. (2022) · Synthese · doi:10.1007/s11229-022-03480-w · Cited by 62
“This has been referred to as ‘intrinsic motivation’ (Barto et al., 2013; Oudeyer & Kaplan, 2007; Schmidhuber, 2010) as well as the drive to seek ‘novelty’ (Schwartenbeck et al., 2019), but more generally involves a drive to learn what will be observed when visiting an unfamiliar state.”
BackgroundMaximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning
Pitis, S., Chan, H., Zhao, S., Stadie, B. C., & Ba, J. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02832 · Cited by 36
“, 2019) or grant the agent additional reward based on some measure of novelty, surprise or learning progress (Kolter & Ng, 2009; Schmidhuber, 2010; Lopes et al., 2012; Bellemare et al., 2016; Ostrovski et al., 2017; Tang et al., 2017; Pathak et al., 2017; Burda et al., 2019).”
BackgroundValues of User Exploration in Recommender Systems
Chen, M., Wang, Y., Xu, C., Le, Y., Sharma, M., Richardson, L. F., Wu, S.-L., & Ed, H. (2021) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3460231.3474236 · Cited by 86
“Schmidhuber [49] formally captures the theory of creativity, fun”
BackgroundWang, R., Lehman, J., Rawal, A., Zhi, J., Li, Y., Clune, J., & Stanley, K. O. (2020) · arXiv · doi:10.48550/arxiv.2003.08536 · Cited by 53
“, 2016) and similarly-oriented work in intrinsic motivation in RL (Bellemare et al., 2016; Oudeyer & Kaplan, 2009; Schmidhuber, 2010).”
BackgroundEvaluation of Musical Creativity and Musical Metacreation Systems
Agres, K., Forth, J., & Wiggins, G. A. (2016) · Computers in Entertainment · doi:10.1145/2967506 · Cited by 48
“No summary of information theoretic approaches to creative cognition would be complete without mention of the work of J¨urgen Schmidhuber—in particular, his Formal Theory of Creativity, Fun, and Intrinsic Motivation [Schmidhuber 2010].”
BackgroundAn Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey
Aubret, A., Matignon, L., & Hassas, S. (2023) · Entropy · doi:10.3390/e25020327 · Cited by 52
Zhou, Q., Zhang, H., & Li, F. (2024) · Education Sciences · doi:10.3390/educsci14060664 · Cited by 70
Continual curiosity-driven skill acquisition from high-dimensional video inputs for humanoid robots
Kompella, V. R., Stollenga, M., Luciw, M., & Schmidhuber, J. (2017) · Artificial Intelligence · doi:10.1016/j.artint.2015.02.001 · Cited by 52
Self-citationOn Reward-Free Reinforcement Learning with Linear Function ApproximationPreprint
Wang, R., Du, S. S., Yang, L. F., & Salakhutdinov, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.11274 · Cited by 121
Learning tactile skills through curious exploration
Pape, L., Oddo, C. M., Controzzi, M., Cipriani, C., Förster, A., Carrozza, M. C., & Schmidhuber, J. (2012) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2012.00006 · Cited by 51
“A major realization of Schmidhuber (2010), is that curious behavior should not direct the agent toward just any unknown or unexplored part of its environment, but to those parts where it expects to learn additional patterns or regularities.”
BackgroundMethodInfluentialSelf-citationNovelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making
He, X., Modirshanechi, A., Lehmann, M. P., Gerstner, W., & Herzog, M. H. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009070 · Cited by 62
“Orthogonal to these measures, enlightenment surprise measures how much an event changes our model of the world and, as a consequence, our expectations; well-known examples are Bayesian surprise [12, 61–64] and compression surprise [12, 63].”
InfluentialCuriosity-driven exploration: foundations in neuroscience and computational modeling
Modirshanechi, A., Kondrakiewicz, K., Gerstner, W., & Haesler, S. (2023) · Trends in Neurosciences · doi:10.1016/j.tins.2023.10.002 · Cited by 61
“Consistent with brain imaging and neurophysiology fi ndings, theories of curiosity often interpret curiosity-driven behaviors as actions driven by an intrinsic reward -like signal (i.e., a signal generated inside the brain) [4,15,92].”
Exploration strategies in developmental robotics: A unified probabilistic framework
Moulin-Frier, C., & Oudeyer, P. (2013) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2013.6652535 · Cited by 59
Universal psychometrics: Measuring cognitive abilities in the machine kingdom
Hernández‐Orallo, J., Dowe, D. L., & Hernández‐Lloreda, M. V. (2013) · Cognitive Systems Research · doi:10.1016/j.cogsys.2013.06.001 · Cited by 59
“Using this theory, there are some proposals for more challenging abilities, such as creativity (Dowe, 2008; Schmidhuber, 2010, sec.”
BackgroundA unified model of the joint development of disparity selectivity and vergence control
Zhao, Y., Rothkopf, C. A., Triesch, J., & Shi, B. E. (2012) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2012.6400876 · Cited by 53
“One path towards a more integrated approach may be found in the ideas of artificial curiosity [4].”
BackgroundLearning to Explore via Meta-Policy Gradient
Xu, T.-B., Liu, Q., Zhao, L., & Peng, J. (2018) · International Conference on Machine Learning · Cited by 69
Evolving Deep LSTM-based Memory Networks using an Information Maximization Objective
Rawal, A., & Miikkulainen, R. P. (2016) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2908812.2908941 · Cited by 48
Ersen, M., Öztop, E., & Sarıel, S. (2017) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2016.2616538 · Cited by 55
“Another article introduces the continual curiositydriven skill acquisition (CCSA) frame work [87] for realizing intrinsically motivated [92] robots that continuously acquire new skills by reusing already learned ones.”
BackgroundCuriosity-Driven Experience Prioritization via Density Estimation
Zhao, R., & Tresp, V. (2019) · arXiv · doi:10.48550/arxiv.1902.08039 · Cited by 47
“Curiosity-driven exploration is a well-studied topic in reinforcement learning [33, 34, 43, 44, 48].”
BackgroundAesthetics and predictive processing: grounds and prospects of a fruitful encounter
Frascaroli, J., Leder, H., Brattico, E., & de Cruys, S. V. (2023) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2022.0410 · Cited by 54
“In this respect, the PP story about aesthetics is just confirming certain long-standing hypotheses about the kind of experiences that facilitate learning and human flourishing [11,139 – 141].”
BackgroundAn intrinsically-motivated schema mechanism to model and simulate emergent cognition
Georgeon, O. L., & Ritter, F. E. (2011) · Cognitive Systems Research · doi:10.1016/j.cogsys.2011.07.003 · Cited by 53
“The second approach implements intrinsic motivation as curiosity and search for novelty (Blank, Kumar, Meeden, & Marshall, 2005; Oudeyer & Kaplan, 2007; Schmidhuber, 2010).”
BackgroundSurprise Search: Beyond Objectives and Novelty
Gravina, D., Liapis, A., & Yannakakis, G. N. (2016) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2908812.2908817 · Cited by 61
Generative models for active vision
Parr, T., Sajid, N., Da Costa, L., Mirza, M. B., & Friston, K. J. (2021) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2021.651432 · Cited by 46
“In short, we have to plan our visual palpation of the world in a way that allows us to construct a scene in our heads that best predicts “what would happen if I looked over there” ( Hassabis and Maguire, 2007; Schmidhuber, 2010; Zeidman et al., 2015; Mirza et al., 2016).”
BackgroundMethodOrder and change in art: towards an active inference account of aesthetic experience
de Cruys, S. V., Frascaroli, J., & Friston, K. J. (2023) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2022.0411 · Cited by 57
“Recent computational theories of curiosity describe it as expected uncertainty resolution, equivalently known as expected information gain or learning progress [18 – 20].”
BackgroundCreativity, information, and consciousness: The information dynamics of thinking
Wiggins, G. A. (2018) · Physics of Life Reviews · doi:10.1016/j.plrev.2018.05.001 · Cited by 55
NROWAN-DQN: A stable noisy network with noise reduction and online weight adjustment for exploration
Han, S. D., Zhou, W., Lu, J., Liu, J., & Lü, S. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117343 · Cited by 52
First experiments with PowerPlay
Srivastava, R. K., Steunebrink, B. R., & Schmidhuber, J. (2013) · Neural Networks · doi:10.1016/j.neunet.2013.01.022 · Cited by 49
Self-citationBeBold: Exploration Beyond the Boundary of Explored RegionsPreprint
Zhang, T., Xu, H., Wang, X., Wu, Y., Keutzer, K., Gonzalez, J. E., & Tian, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.08621 · Cited by 49
“Recent approaches have proposed to use intrinsic rewards (IR) (Schmidhuber, 2010) to motivate agents for exploration before any extrinsic rewards are obtained.”
BackgroundSchlesinger, M., & Amso, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00802 · Cited by 44
“…we remain agnostic to the specific cognitive structures or architectures exploited by human infants during visual exploration, a more elegant solution on the computational side would be to employ a single, unified predictor that learns over a range of sampling strategies (e.g., Schmidhuber, 2010).”
BackgroundEmergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning
Sharma, A., Ahn, M., Levine, S., Kumar, V., Hausman, K., & Gu, S. (2020) · Robotics: Science and Systems · doi:10.15607/rss.2020.xvi.053 · Cited by 57
Curiosity in exploring chemical spaces: intrinsic rewards for molecular reinforcement learning
Thiede, L., Krenn, M., Nigam, A., & Aspuru‐Guzik, A. (2022) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ac7ddc · Cited by 54
“Curiosity falls under the wider category of intrinsic motivation techniques [18, 20, 21], which are loosely modeled after human curiosity.”
BackgroundCuriosity: From psychology to computation
Wu, Q., & Miao, C. (2013) · ACM Computing Surveys · doi:10.1145/2543581.2543585 · Cited by 58
Aesthetic experience models human learning
Starr, G. G. (2023) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2023.1146083 · Cited by 45
“This means that pleasure correlates with an outcome— a perception—that is predicted by past experience (Schmidhuber, 2010).”
BackgroundInfluentialSymmetry-Based Representations for Artificial and Biological General Intelligence
Higgins, I., Racanière, S., & Rezende, D. J. (2022) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2022.836498 · Cited by 62
“…perspective, such representations can be viewed as the simplest (in the context of Solomonoff induction; Solomonoff, 1964) and the most informative representations of the input to support the most likely future tasks (MacKay, 1995, 2003;Wallace and Dowe, 1999; Hutter, 2004; Schmidhuber, 2010).”
BackgroundActive World Model Learning with Progress CuriosityPreprint
Kim, K., Sano, M., Freitas, J. D., Haber, N., & Yamins, D. L. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07853 · Cited by 90
“As a result, the curious agent suffers from the white noise problem [Schmidhuber, 2010], i.e it endlessly fixates on unlearnable stimuli.”
BackgroundIntelligent soft matter: towards embodied intelligence
Baulin, V. A., Giacometti, A., Fedosov, D. A., Ebbens, S. J., Varela-Rosales, N. R., Feliu, N., Chowdhury, M., Hu, M., Füchslin, R. M., Dijkstra, M., et al. (2025) · Soft Matter · doi:10.1039/d5sm00174a · Cited by 54
Learning in Volatile Environments With the Bayes Factor Surprise
Liakoni, V., Modirshanechi, A., Gerstner, W., & Brea, J. (2021) · Neural Computation · doi:10.1162/neco_a_01352 · Cited by 46
“The Bayesian Surprise SBa (Itti & Baldi, 2006; Schmidhuber, 2010; Storck et al., 1995) and the Confidence Corrected Surprise SCC (Faraji et al., 2018) are two other measures of surprise in neuroscience.”
BackgroundInfluentialAnimal-in-the-Loop: Using Interactive Robotic Conspecifics to Study Social Behavior in Animal Groups
Landgraf, T., Gebhardt, G. H. W., Bierbach, D., Romańczuk, P., Musiolek, L. M., Hafner, V. V., & Krause, J. (2020) · Annual Review of Control Robotics and Autonomous Systems · doi:10.1146/annurev-control-061920-103228 · Cited by 50
Friston, K. J., Costa, L. D., Tschantz, A., Kiefer, A. B., Salvatori, T., Neacsu, V., Koudahl, M., Heins, C., Sajid, N., Marković, D., et al. (2024) · Biological Psychology · doi:10.1016/j.biopsycho.2024.108891 · Cited by 46
Curiosity-driven phonetic learning
Moulin-Frier, C., & Oudeyer, P.-Y. (2012) · 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL) · doi:10.1109/devlrn.2012.6400583 · Cited by 36
“been transposed in curiosity-driven learning machines [9]– [11] and robots [5], [6] and shown to yield highly efficient learning of inverse models in high-dimensional redundant sensorimotor spaces [5].”
BackgroundNot by transmission alone: the role of invention in cultural evolution
Perry, S., Carter, A. J., Smolla, M., Akçay, E., Nöbel, S., Foster, J. G., & Healy, S. D. (2021) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2020.0049 · Cited by 46
“Such theories emphasize novelty-seeking, whether through the recognition (and avoidance) of experiential ‘ ruts ’ [17] or through intrinsic reward from encountering new experiences that can be profitably ‘ compressed ’ through learning [52].”
BackgroundJackson, L. A., & Games, A. (2015) · Elsevier eBooks · doi:10.1016/b978-0-12-801462-2.00001-1 · Cited by 45
Machine Autonomy: Definition, Approaches, Challenges and Research Gaps
Ezenkwu, C. P., & Starkey, A. J. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-22871-2_24 · Cited by 47
Gamification inducing creative ideation: a parallel mediation model
Nivedhitha, K. S., & Manzoor, A. K. S. (2019) · Behaviour and Information Technology · doi:10.1080/0144929x.2019.1635646 · Cited by 44
“In support of transcendent experience, Russ (1999) put forward the positive e ff ect of emotional expressiveness on creativity, and Schmidhuber (2010) proposed an interesting model that illustrates that ‘ wow e ff ects, ’ (such as transcendent experience) intrinsically motivate even robots/agents to…”
BackgroundSafron, A., Çatal, O., & Verbelen, T. (2022) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2022.787659 · Cited by 35
“After an initial period of relatively ambiguous exploration, the formation of a causal account (or trajectory through a concept space/graph) would allow for a rapid decrease in prediction-error (Joffily and Coricelli, 2013), or increase in compression (Schmidhuber, 2010).”
BackgroundHierarchical curiosity loops and active sensing
Gordon, G., & Ahissar, E. (2012) · Neural Networks · doi:10.1016/j.neunet.2012.02.024 · Cited by 42
“Each hierarchical loop architecture is composed of learners (predictors), intrinsic rewards and RL controllers, namely, critics and actors (Schmidhuber, 2010).”
BackgroundMethodInfluentialMotivated to learn: An account of explanatory satisfaction
Liquin, E. G., & Lombrozo, T. (2021) · Cognitive Psychology · doi:10.1016/j.cogpsych.2021.101453 · Cited by 45
““; see Liquin & Lombrozo, 2020a; Oudeyer et al., 2007; Schmidhuber, 2010), and the other about the value of available informants (e.g., “How good are explanations from this source?”).”
BackgroundEvolved to adapt: A computational approach to animal innovation and creativity
Kolodny, O., Edelman, S., & Lotem, A. (2015) · Current Zoology · doi:10.1093/czoolo/61.2.350 · Cited by 39
“We suggest briefly that like other emotional states, it may be related to drawing attention and assigning more weight in memory to rare but important discoveries (see, e.g., Lotem and Halpern, 2012; Schmidhuber, 2010; 2013).”
BackgroundA theory and methodology to quantify knowledge
Fanelli, D. (2019) · Royal Society Open Science · doi:10.1098/rsos.181055 · Cited by 44
“The sensual and 1807 intellectual pleasure that humans gain from music and art, for example, seems to derive from an 1808 optimal balance between perception of structure (pattern that generates predictions and 1809 expectations) and perception of novelty (which stimulates interest by presenting new and knowable 1810 i…”
BackgroundInteractive Language Learning by Question Answering
Yuan, X., Côté, M.-A., Fu, J., Lin, Z., Pal, C., Bengio, Y., & Trischler, A. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1280 · Cited by 45
Meaning from movement and stillness: Signatures of coordination dynamics reveal infant agency
Sloan, A., Jones, N. A., & Kelso, J. A. S. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2306732120 · Cited by 37
Oversampled and undersolved: Depressive rumination from an active inference perspective
Berg, M., Feldmann, M., Kirchner, L., & Κube, T. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104873 · Cited by 38
Hangl, S., Dunjko, V., Briegel, H. J., & Piater, J. (2020) · Frontiers in Robotics and AI · doi:10.3389/frobt.2020.00042 · Cited by 38
“Schmidhuber (2010) provides a sophisticated summary of work on intrinsic motivation and embeds the idea into a general framework.”
BackgroundOne Big Net For EverythingPreprint
Schmidhuber, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.08864 · Cited by 38
“Hence, another goal of ONE can be to compress ONE’s entire growing interaction history of all failed and successful trials [70, 73], e.g., through neural predictive coding [61, 78].”
BackgroundSelf-citationCausally Correct Partial Models for Reinforcement Learning
Rezende, D. J., Danihelka, I., Papamakarios, G., Ke, N. R., Jiang, R., Weber, T., Gregor, K., Merzić, H., Viola, F., Wang, J., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02836 · Cited by 15
“Within MBRL, commonly explored methods include action-conditional, next-step models (Oh et al., 2015; Ha & Schmidhuber, 2018; Chiappa et al., 2017; Schmidhuber, 2010; Xie et al., 2016; Deisenroth & Rasmussen, 2011; Lin & Mitchell, 1992; Li et al., 2015; Diuk et al., 2008; Igl et al., 2018; Ebert et…”
MethodReward maximization through discrete active inference
Da Costa, L., Sajid, N., Parr, T., Friston, K., & Smith, R. (2023) · Neural Computation · doi:10.1162/neco_a_01574 · Cited by 35
“In finite horizon MDPs, temporal discounting is not necessary so we set γ = 1 (see Schmidhuber, 2006, 2010).”
BackgroundHuman inference reflects a normative balance of complexity and accuracy
Tavoni, G., Doi, T., Pizzica, C., Balasubramanian, V., & Gold, J. I. (2022) · Nature Human Behaviour · doi:10.1038/s41562-022-01357-z · Cited by 35
“Finally, the leading-order term of 〈NS〉 (2N2) arises because the computation of the adaptive p(ri∣x1:t) requires maintaining in memory, at every iteration, the N × N matrices of the transition probabilities p(ri∣rj, CP) and p(ri∣rj, no CP) (equations (24) through (27)).”
BackgroundDu, K.-L., Zhang, R., Jiang, B., Zeng, J., & Lu, J. (2025) · Mathematics · doi:10.3390/math13030451 · Cited by 44
Technological Singularity: What Do We Really Know?
Potapov, A. S. (2018) · Information · doi:10.3390/info9040082 · Cited by 35
“I will not try to dispel these doubts here, but simply note that there are different opinions on this topic, and that theories of artificial creativity, curiosity, and fun exist (e.g., [21]).”
BackgroundCuriosity-Driven Exploration via Latent Bayesian Surprise
Mazzaglia, P., Çatal, O., Verbelen, T., & Dhoedt, B. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20743 · Cited by 60
“However, searching for less predictable states suffers from the ‘NoisyTV problem’, where watching a screen outputting white random noise appears more interesting than other exploratory behaviours (Schmidhuber 2010).”
BackgroundMethodGrounding Language in Play.Preprint
Lynch, C., & Sermanet, P. (2020) · arXiv (Cornell University) · Cited by 44
Interactional Motivation in artificial systems: Between extrinsic and intrinsic motivation
Georgeon, O. L., Marshall, J. B., & Gay, S. L. (2012) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2012.6400833 · Cited by 35
A Minimal Theory of Creative Ability
Stevenson, C. E., Baas, M., & van der Maas, H. L. J. (2021) · Journal of Intelligence · doi:10.3390/jintelligence9010009 · Cited by 42
ECA: An enactivist cognitive architecture based on sensorimotor modeling
Georgeon, O. L., Marshall, J. B., & Manzotti, R. (2013) · Biologically Inspired Cognitive Architectures · doi:10.1016/j.bica.2013.05.006 · Cited by 35
“More broadly, autotelic motivation falls within the area of intrinsic motivation (e.g., Oudeyer, Kaplan, & Hafner, 2007; Blank, Kumar, Meeden, & Marshall, 2005; Schmidhuber, 2010) because the agent’s preferences are defined independently of any reference to the environment’s states.”
Stanton, C., & Clune, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.00553 · Cited by 16
Surprise and novelty in the brain
Modirshanechi, A., Becker, S., Brea, J., & Gerstner, W. (2023) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2023.102758 · Cited by 39
“…full belief p ( t ) ( q t þ 1 ) about the hidden parameter set q t þ 1 and requires even more information than the full distribution p ( t ) ( y t þ 1 | x t þ 1 ); an example is the Bayesian surprise s t þ 1 = D KL ( p ( t ) , p ( t þ 1) ), where D KL denotes Kullback-Leibler divergence [31,32].”
Learning to recognize objects through curiosity-driven manipulation with the iCub humanoid robot
Nguyen, S. M., Ivaldi, S., Lyubova, N., Droniou, A., Gerardeaux-Viret, D., Filliat, D., Padois, V., Sigaud, O., & Oudeyer, P. (2013) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2013.6652525 · Cited by 37
“This inspired the creation of fully autonomous robots with meta-exploration mechanisms monitoring the evolution of learning performances [35], [36], [37], with heuristics defining the notion of interest used in an active learning framework [38], [39].”
Background
Metadata
- Authors
- Schmidhuber, J.
- Affiliations
- Dalle Molle Institute for Artificial Intelligence Research (CH)
- Corresponding author
- Schmidhuber, Jürgen
- ORCID
- Schmidhuber, Jürgen
- Venue
- IEEE Transactions on Autonomous Mental Development
- Published
- 2010-09
- Volume
- 2
- Issue
- 3
- Pages
- 230-247
- ISSN
- 1943-0604 (ISSN-L)
- 1943-0612
- OpenAlex
- W2034806191
- Cited by
- 747
- References
- 126
- Citations by year
- 2026: 29 · 2025: 64 · 2024: 46 · 2023: 50 · 2022: 60 · 2021: 69 · 2020: 92 · 2019: 72 · 2018: 55 · 2017: 37 · 2016: 26 · 2015: 24 · 2014: 30 · 2013: 28 · 2012: 41
- FWCI
- 31.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Autonomous Mental Development (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Creativity in Education and Neuroscience
- Keywords
- intrinsic motivation, creativity, fun, intrinsic rewards, active learning (machine learning), novelty seeking, surprise, pattern discovery, data compression, online prediction, autonomous agents, robots, aesthetics, humor, intelligence, limited computational resources, continuous-time formulation, discrete-time formulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.