The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
A Possibility for Implementing Curiosity and Boredom in Model-Building Neural Controllers
Open accessAbstract
This paper introduces a framework for `curious neural controllers' which employ an adaptive world model for goal directed on-line learning. First an on-line reinforcement learning algorithm for autonomous `animats' is described. The algorithm is based on two fully recurrent `self-supervised' continually running networks which learn in parallel. One of the networks learns to represent a complete model of the environmental dynamics and is called the `model network'. It provides complete `credit assignment paths' into the past for the second network which controls the animats physical actions in a possibly reactive environment. The animats goal is to maximize cumulative reinforcement and minimize cumulative `pain'. The algorithm has properties which allow to implement something like the desire to improve the model network's knowledge about the world. This is related to curiosity. It is described how the particular algorithm (as well as similar model-building algorithms) may be augmented ...
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The truck backer-upper: an example of self-learning in neural networks
Nguyen, & Widrow (1989) · International Joint Conference on Neural Networks · doi:10.1109/ijcnn.1989.118723 · Cited by 449
Experimental Analysis of the Real-time Recurrent Learning Algorithm
WILLIAMS, R. J., & ZIPSER, D. (1989) · Connection Science · doi:10.1080/09540098908915631 · Cited by 292
Supervised learning and systems with excess degrees of freedom
Jordan, M. I. (1988) · Cited by 180
An on-line algorithm for dynamic reinforcement learning and planning in reactive environments
Schmidhuber, J. (1990) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1990.137723 · Cited by 130
Self-citationBackpropagation and neurocontrol: a review and prospectus
Werbos, P. J. (1989) · doi:10.1109/ijcnn.1989.118583 · Cited by 129
Static and Dynamic Error Propagation Networks with Application to Speech Coding
Robinson, A. J., & Fallside, F. (1987) · Cited by 79
On the use of backpropagation in associative reinforcement learning
Williams (1988) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1988.23856 · Cited by 72
Dynamic Reinforcement Driven Error Propagation Networks with Application to Game Playing
Robinson, T., & Fallside, F. (1989) · eScholarship (California Digital Library) · Cited by 18
Cited by
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733
“UCB can also be considered one way of implementing intrinsic motivation, which is a general concept that advocates decreasing uncertainty/making progress in learning about the environment [101].”
A Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815
“Measuring learning progress instead of prediction error has been proposed in the past as one solution (Schmidhuber, 1991).”
BackgroundInfluentialA Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
Gui, J., Sun, Z., Wen, Y., Tao, D., & Ye, J. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3130191 · Cited by 1,231
Meta-Learning in Neural Networks: A Survey
Hospedales, T., Antoniou, A., Micaelli, P., & Storkey, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05439 · Cited by 886
Meta-Learning in Neural Networks: A Survey
Hospedales, T. M., Antoniou, A., Micaelli, P., & Storkey, A. J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3079209 · Cited by 886
The Effort Paradox: Effort Is Both Costly and Valued
Inzlicht, M., Shenhav, A., & Olivola, C. Y. (2018) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.01.007 · Cited by 863
Formal theory of creativity, fun, and intrinsic motivation (1990–2010)
Schmidhuber, J. (2010) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2010.2056368 · Cited by 747
Self-citation(2016) · neural information processing systems · Cited by 735
Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective
Botvinick, M. M., Niv, Y., & Barto, A. G. (2009) · Cognition · doi:10.1016/j.cognition.2008.08.011 · Cited by 646
A survey and critique of multiagent deep reinforcement learning
Hernandez-Leal, P., Kartal, B., & Taylor, M. E. (2019) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-019-09421-1 · Cited by 671
Intrinsically Motivated Reinforcement Learning
Singh, S., Barto, A. G., & Chentanez, N. (2005) · doi:10.21236/ada440280 · Cited by 610
Curious model-building control systems
Schmidhuber, J. (1991) · [Proceedings] 1991 IEEE International Joint Conference on Neural Networks · doi:10.1109/ijcnn.1991.170605 · Cited by 541
Self-citationExploration in deep reinforcement learning: A survey
Ladosz, P., Weng, L., Kim, M., & Oh, H. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.003 · Cited by 525
“The simplest method of this type was described in [10, 11].”
BackgroundMethodExploration by Random Network Distillation
Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018) · arXiv · doi:10.48550/arxiv.1810.12894 · Cited by 527
“Another class of exploration methods rely on errors in predicting dynamics (Schmidhuber, 1991b; Stadie et al., 2015; Achiam & Sastry, 2017; Pathak et al., 2017; Burda et al., 2018).”
BackgroundMethodInfluentialArtificial General Intelligence: Concept, State of the Art, and Future Prospects
Goertzel, B. (2014) · Journal of Artificial General Intelligence · doi:10.2478/jagi-2014-0001 · Cited by 524
Active learning of inverse models with intrinsically motivated goal exploration in robots
Baranes, A., & Oudeyer, P.-Y. (2013) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2012.05.008 · Cited by 455
Emergent Tool Use From Multi-Agent Autocurricula
Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019) · arXiv · doi:10.48550/arxiv.1909.07528 · Cited by 424
“Another paradigm are transition-based methods, in which agents are rewarded for high prediction error in a learned forward or inverse dynamics model [42, 43, 44, 45, 46, 47, 48, 49].”
BackgroundIntrinsically Motivated Reinforcement Learning: An Evolutionary Perspective
Singh, S., Lewis, R. L., Barto, A. G., & Sorg, J. (2010) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2010.2051031 · Cited by 417
Large-Scale Study of Curiosity-Driven Learning
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., & Efros, A. A. (2018) · arXiv · doi:10.48550/arxiv.1808.04355 · Cited by 401
“Examples of intrinsic reward include “curiosity” [11, 22, 27, 35, 40] which uses prediction error as reward signal, and “visitation counts” [3, 20, 24, 30] which discourages the agent from revisiting the same states.”
BackgroundDynamic Task Prioritization for Multitask Learning
Guo, M., Haque, A., Huang, D.-A., Yeung, S., & Fei-Fei, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01270-0_17 · Cited by 356
Murayama, K. (2022) · Psychological Review · doi:10.1037/rev0000349 · Cited by 304
Developmental robotics, optimal artificial curiosity, creativity, music, and the fine arts
Schmidhuber, J. (2006) · Connection Science · doi:10.1080/09540090600768658 · Cited by 268
Self-citationLipton, Z. C., & Steinhardt, J. (2019) · Queue · doi:10.1145/3317287.3328534 · Cited by 231
Go-Explore: a New Approach for Hard-Exploration Problems
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J. (2019) · arXiv · doi:10.48550/arxiv.1901.10995 · Cited by 205
Ha, D., & Schmidhuber, J. (2018) · Zenodo · doi:10.5281/zenodo.1207631 · Cited by 187
Self-citationWho Gets Credit for AI-Generated Art?
Epstein, Z., Levine, S., Rand, D. G., & Rahwan, I. (2020) · iScience · doi:10.1016/j.isci.2020.101515 · Cited by 157
Planning to Explore via Self-Supervised World Models
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., & Pathak, D. (2020) · arXiv · doi:10.48550/arxiv.2005.05960 · Cited by 129
EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
Fu, J., Co-Reyes, J. D., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.01260 · Cited by 95
Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scale
Ramrakhya, R., Undersander, E., Batra, D., & Das, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00511 · Cited by 90
Neural circuitry of information seeking
Bromberg-Martin, E. S., & Monosov, I. E. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.07.006 · Cited by 89
Intrinsic motivations and open-ended development in animals, humans, and robots: an overview
Baldassarre, G., Stafford, T., Mirolli, M., Redgrave, P., Ryan, R. M., & Barto, A. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00985 · Cited by 73
A Survey of Deep Learning Applications to Autonomous Vehicle Control
Kuutti, S., Bowden, R., Jin, Y., Barber, P., & Fallah, S. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2962338 · Cited by 96
“Therefore, intrinsic motivation is implemented in the system, encouraging the agent to learn about its environment, whilst exploitation can be taken advantage of in states which have already been explored adequately [51], [117]–[119].”
BackgroundInfluence-Based Multi-Agent Exploration
Wang, T., Wang, J., Wu, Y., & Zhang, C. (2019) · arXiv · doi:10.48550/arxiv.1910.05512 · Cited by 69
Schmidhuber, J. (2003) · Natural Computing Series · doi:10.1007/978-3-642-18965-4_23 · Cited by 65
Self-citationSemantic Curiosity for Active Visual Learning
Chaplot, D. S., Jiang, H., Gupta, S., & Gupta, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_19 · Cited by 64
The alignment problem from a deep learning perspective
Ngo, R., Chan, L., & Mindermann, S. (2022) · arXiv · doi:10.48550/arxiv.2209.00626 · Cited by 68
“However, this will become signifi-cantly more difficult as policies develop situational awareness , as described below.”
BackgroundImproving Playtesting Coverage via Curiosity Driven Reinforcement Learning Agents
Gordillo, C., Bergdahl, J., Tollmar, K., & Gisslen, L. (2021) · 2021 IEEE Conference on Games (CoG) · doi:10.1109/cog52621.2021.9619048 · Cited by 50
Curiosity-Driven Experience Prioritization via Density Estimation
Zhao, R., & Tresp, V. (2019) · arXiv · doi:10.48550/arxiv.1902.08039 · Cited by 47
“Curiosity-driven exploration is a well-studied topic in reinforcement learning [33, 34, 43, 44, 48].”
BackgroundButz, M. V. (2021) · KI - Künstliche Intelligenz · doi:10.1007/s13218-021-00705-x · Cited by 42
“Second, a motivational system maintains a healthy but complex balance between epistemic- and homeostasis-oriented inductive processing biases [19, 25, 69, 78, 90, 92].”
BackgroundLearning Periodic Tasks from Human Demonstrations
Yang, J., Zhang, J., Settle, C., Rai, A., Antonova, R., & Bohg, J. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9812402 · Cited by 21
Kadi, H. A., & Terzić, K. (2023) · Sensors · doi:10.3390/s23052389 · Cited by 27
“The DRL community has focused on optimistic methods, including intrinsic motivation [199,200], MaxEnt-RL (Section 5.5.1) and sampling-based methods [190,201,202] based on TS for tackling exploration in complex MDP/POMDP settings.”
MethodInfluentialExploration and preference satisfaction trade-off in reward-free learning
Sajid, N., Tigas, P., Zakharov, A., Fountas, Z., & Friston, K. (2021) · arXiv · doi:10.48550/arxiv.2106.04316 · Cited by 11
“Over the years, a variety of intrinsic motivation methods have been proposed, largely focusing on exploration, based on information gain [28, 62], prediction error [1, 43, 59], novelty search [37, 54], curiosity [48, 49], entropy [20, 36], or empowerment [34, 40].”
BackgroundCraftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning
Matthews, M., Beukman, M., Ellis, B., Samvelyan, M., Jackson, M., Coward, S., & Foerster, J. (2024) · arXiv · doi:10.48550/arxiv.2402.16801 · Cited by 4
“One method to overcome these difficulties is through the idea of an intrinsic reward (Schmidhuber, 1991; Oudeyer et al., 2007; Barto, 2013; Bellemare et al., 2016), in addition to the extrinsic reward received from the environment.”
MethodA Study of Global and Episodic Bonuses for Exploration in Contextual MDPs
Henaff, M., Jiang, M., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2306.03236 · Cited by 1
“Exploration in singleton MDPs is a well-studied problem in RL (Sutton & Barto, 2018; Schmidhuber, 1991; Oudeyer et al., 2007; Oudeyer & Kaplan, 2009).”
BackgroundImproving Environment Novelty Quantification for Effective Unsupervised Environment Design
Teoh, J., Li, W., & Varakantham, P. (2025) · arXiv
“Curiosity-driven Approaches in RL CENIE and curiosity-driven RL [47, 50] share a conceptual similarity in leveraging novelty or unfamiliarity to guide learning.”
Improving Environment Novelty Quantification for Effective Unsupervised Environment Design
Teoh, J., Li, W., & Varakantham, P. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-4300
Metadata
- Authors
- Schmidhuber, J.
- Venue
- From Animals to Animats
- Publisher
- The MIT Press
- Published
- 1991-02-04
- Pages
- 222-228
- OpenAlex
- W172298727
- Cited by
- 421
- References
- 8
- FWCI
- 1.37
- Type
- book chapter
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.