Planning to be surprised: optimal bayesian exploration in dynamic environments
Open accessIntegrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This intrinsic value maximisation underwrites Bayesian experimental design, active learning, intrinsic motivation, and artificial curiosity.
“This underwrites Bayesian experimental design [134] and active learning in statistics [135], intrinsic motivation and artificial curiosity in machine learning and robotics [136, 137, 138, 139, 140].”
From The free energy principle made simpler but not too simple (2023) · cites this paper as [139]
Adaptive sampling methods are also known in reinforcement learning that can achieve optimal Bayesian exploration of Markov Decision Process environments.
“Adaptive sampling methods are also known in reinforcement learning that can achieve optimal Bayesian exploration of Markov Decision Process environments (Sun et al., 2011; Guez et al., 2012).”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Sun et al., 2011
Cites
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791 · see citing works
Fedorov, V. V., Studden, W. J., & Klimko, E. M. (1972) · Cited by 2,057 · see citing works
Theory of Optimal Experiments.
Herzberg, A. M., Wynn, H. P., Fedorov, V. V., Studden, W. J., & Klimko, E. M. (1972) · Biometrika · doi:10.2307/2334826 · Cited by 1,896 · cited 2× · see citing works
“In particular, measuring learning progress using KL divergence dates back to the 50’s [2,4].”
BackgroundBayesian Experimental Design: A Review
Chaloner, K. M., & Verdinelli, I. (1995) · Statistical Science · doi:10.1214/ss/1177009939 · Cited by 1,843 · cited 1× · see citing works
“The idea of actively selecting queries to accelerate learning process has a long history [1, 2, 7], and has received a lot of attention in recent decades, primarily in the context of active learning [8] and artificial curiosity [6].”
BackgroundOn a measure of the information provided by an experiment
Lindley, D. V. (1956) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177728069 · Cited by 1,509 · cited 1× · see citing works
“In particular, measuring learning progress using KL divergence dates back to the 50’s [2,4].”
BackgroundFormal theory of creativity, fun, and intrinsic motivation (1990–2010)
Schmidhuber, J. (2010) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2010.2056368 · Cited by 747 · cited 2× · see citing works
“Conceptually, however, this work is closely connected to artificial curiosity and intrinsically motivated reinforcement learning [6,7,9] for agents that actively explore the environment without an external reward signal.”
BackgroundCurious model-building control systems
Schmidhuber, J. (1991) · [Proceedings] 1991 IEEE International Joint Conference on Neural Networks · doi:10.1109/ijcnn.1991.170605 · Cited by 733 · cited 2× · see citing works
“Conceptually, however, this work is closely connected to artificial curiosity and intrinsically motivated reinforcement learning [6,7,9] for agents that actively explore the environment without an external reward signal.”
BackgroundIntrinsically Motivated Reinforcement Learning
Singh, S. P., Barto, A. G., & Chentanez, N. (2005) · doi:10.21236/ada440280 · Cited by 614 · cited 1× · see citing works
“Conceptually, however, this work is closely connected to artificial curiosity and intrinsically motivated reinforcement learning [6,7,9] for agents that actively explore the environment without an external reward signal.”
BackgroundBayesian Surprise Attracts Human Attention
Itti, L., & Baldi, P. (2005) · neural information processing systems · Cited by 363 · see citing works
Planning to Be Surprised: Optimal Bayesian Exploration in Dynamic EnvironmentsPreprint
Sun, Y., Gomez, F. J., & Schmidhuber, J. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.5708 · Cited by 25 · see citing works
Self-citation
Flagged first, then by citations
1 of the 11 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Curiosity-driven Exploration in Deep Reinforcement Learning via Bayesian Neural NetworksPreprint
Houthooft, R., Chen, X., Duan, Y., Schulman, J. D., de Turck, F., & Abbeel, P. (2016) · arXiv.org · doi:10.48550/arxiv.1605.09674 · Cited by 81 · see citing works
#Exploration: A Study of Count-Based Exploration for Deep Reinforcement LearningPreprint
Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04717 · Cited by 486 · see citing works
“Bayesian RL methods (Kolter & Ng, 2009; Guez et al., 2014; Sun et al., 2011; Ghavamzadeh et al., 2015), which keep track of a distribution over MDPs, are an alternative to optimism-based methods.”
MethodGerstner, W., Lehmann, M., Liakoni, V., Corneil, D., & Brea, J. (2018) · Frontiers in Neural Circuits · doi:10.3389/fncir.2018.00053 · Cited by 321 · see citing works
“Theoreticians have studied these ideas in the context of curiosity (Schmidhuber, 2010), information gain during active exploration (Storck et al., 1995; Schmidhuber, 2006; Sun et al., 2011; Little and Sommer, 2013; Friston et al., 2016), and via formal definitions of surprise (Shannon, 1948; Storck et al.”
BackgroundComputational mechanisms of curiosity and goal-directed exploration
Schwartenbeck, P., Passecker, J., Hauser, T. U., FitzGerald, T. H. B., Kronbichler, M., & Friston, K. J. (2019) · eLife · doi:10.7554/elife.41703 · Cited by 250 · see citing works
Planning and navigation as active inference
Kaplan, R., & Friston, K. J. (2018) · Biological Cybernetics · doi:10.1007/s00422-018-0753-2 · Cited by 203 · see citing works
Active inference on discrete state-spaces: A synthesis
Costa, L. D., Parr, T., Sajid, N., Veselic, S., Neacsu, V., & Friston, K. J. (2020) · UCL Discovery (University College London) · Cited by 204 · see citing works
Intrinsic Valuation of Information in Decision Making under Uncertainty
Bennett, D., Bode, S., Brydevall, M., Warren, H. A., & Murawski, C. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005020 · Cited by 186 · see citing works
“In particular, it is possible to reparametrise Eq 6 such that intrinsic valuation of information is expressed in terms of value bonus for information, rather than an aversion to states’ uncertainty (see also Sun, Gomez & Schmidhuber, 2011 [49] for a discussion of conceptual differences between information reward signa…”
BackgroundSentience and the Origins of Consciousness: From Cartesian Duality to Markovian Monism
Friston, K. J., Wiese, W., & Hobson, J. A. (2020) · Entropy · doi:10.3390/e22050516 · Cited by 186 · see citing works
“, expected surprisal or uncertainty), we now have a plausible description of intentional behaviour that will, to all intents and purposes, look like uncertainty resolving, information seeking, epistemic foraging [81,83,84,26,85,86,87,88,89,90].”
BackgroundSelf-Supervised Exploration via DisagreementPreprint
Pathak, D., Gandhi, D., & Gupta, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04161 · Cited by 190 · see citing works
“This prediction error can also be formulated as surprise (Achiam & Sastry, 2017; Schmidhuber, 1991a; Sun et al., 2011).”
BackgroundThe free energy principle made simpler but not too simple
Friston, K., Da Costa, L., Sajid, N., Heins, C., Ueltzhöffer, K., Pavliotis, G. A., & Parr, T. (2023) · Physics Reports · doi:10.1016/j.physrep.2023.07.001 · Cited by 163 · see citing works
“This underwrites Bayesian experimental design [134] and active learning in statistics [135], intrinsic motivation and artificial curiosity in machine learning and robotics [136, 137, 138, 139, 140].”
Planning to Explore via Self-Supervised World Models
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., & Pathak, D. (2020) · arXiv · doi:10.48550/arxiv.2005.05960 · Cited by 129 · see citing works
“Sun et al. (2011) derived a model-based reinforcement learning agent that can optimize the infinite-horizon information gain and experimented with it in tabular settings.”
BackgroundLearning action-oriented models through active inference
Tschantz, A., Seth, A. K., & Buckley, C. L. (2020) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007805 · Cited by 121 · see citing works
“…models to determine epistemic (or intrinsic, information-seeking, uncertainty reducing ) actions that attempt to minimize the model uncertainty in a directed manner (Stadie et al., 2015; Houthooft et al., 2016; Sun et al., 2011; Friston et al., 2015; Burda et al., 2018; Friston et al., 2017).”
BackgroundToward an integration of deep learning and neuroscience
Marblestone, A. H., Wayne, G., & Kording, K. P. (2016) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2016.00094 · Cited by 119 · see citing works
“Adaptive sampling methods are also known in reinforcement learning that can achieve optimal Bayesian exploration of Markov Decision Process environments (Sun et al., 2011; Guez et al., 2012).”
MethodFriston, K. J., Costa, L. D., Hafner, D., Hesp, C., & Parr, T. (2021) · Neural Computation · doi:10.1162/neco_a_01351 · Cited by 123 · see citing works
“This is mathematically the same as expected Bayesian surprise and mutual information that underwrites salience in visual search (Itti and Baldi, 2009; Sun et al., 2011) and the organisation of our visual apparatus (Barlow, 1961; Barlow, 1974; Linsker, 1990; Optican and Richmond, 1987).”
BackgroundSchmidhuber, J. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00313 · Cited by 121 · see citing works
“xp loration seem obvious. One should analyze theoretically and experimentally under which conditions the creation of self-generated tasks can accelerate the solution to externally generated tasks—see [30, 54, 37, 38, 19, 4, 28, 62] for previous simple experimental studies in this vein. 6.1 Self-Reference Through Novel…”
BackgroundResultInfluentialPath integrals, particular kinds, and strange things
Friston, K. J., Costa, L. D., Sakthivadivel, D. A. R., Heins, C., Pavliotis, G. A., Ramstead, M. J. D., & Parr, T. (2023) · Physics of Life Reviews · doi:10.1016/j.plrev.2023.08.016 · Cited by 122 · see citing works
Modern Bayesian Experimental Design
Rainforth, T., Foster, A., Ivanova, D. R., & Smith, F. B. (2024) · Statistical Science · doi:10.1214/23-sts915 · Cited by 110 · see citing works
Evidence for surprise minimization over value maximization in choice behavior
Schwartenbeck, P., FitzGerald, T. H. B., Mathys, C., Dolan, R., Kronbichler, M., & Friston, K. (2015) · Scientific Reports · doi:10.1038/srep16575 · Cited by 96 · see citing works
An Exploration of Embodied Visual Exploration
Ramakrishnan, S. K., Jayaraman, D., & Grauman, K. L. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01437-z · Cited by 101 · see citing works
“Reconstruction-based exploration approaches also relate to classical informationgain based exploration, where the goal is to explore in order to reduce the uncertainty in the agent’s model of the world [13,63,66].”
Representation Wars: Enacting an Armistice Through Active Inference
Constant, A., Clark, A., & Friston, K. J. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.598733 · Cited by 97 · see citing works
“In visual neurosciences, this is called salience and is a key determinant of epistemic foraging or exploratory behavior (Itti and Baldi, 2009; Sun et al., 2011).”
BackgroundFriston, K. J., Sajid, N., Quiroga‐Martinez, D. R., Parr, T., Price, C. J., & Holmes, E. (2020) · Hearing Research · doi:10.1016/j.heares.2020.107998 · Cited by 102 · see citing works
Whence the Expected Free Energy?
Millidge, B., Tschantz, A., & Buckley, C. L. (2021) · Neural Computation · doi:10.1162/neco_a_01354 · Cited by 94 · see citing works
World model learning and inference
Friston, K. J., Moran, R., Nagai, Y., Taniguchi, T., Gomi, H., & Tenenbaum, J. B. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.09.011 · Cited by 89 · see citing works
“…both pragmatic (extrinsic) and epistemic (intrinsic) imperatives for good plans that resolve the exploration–exploitation dilemma (Cohen, McClure, & Yu, 2007; Friston et al., 2015; Schmidhuber, 2006; Still & Precup, 2012; Sun, Gomez, & Schmidhuber, 2011; Tschantz, Baltieri, Seth, & Buckley, 2020).”
Play in predictive minds: A cognitive theory of play.
Andersen, M. M., Kiverstein, J., Miller, M., & Roepstorff, A. (2023) · Psychological Review · doi:10.1037/rev0000369 · Cited by 122 · see citing works
Neural signals encoding shifts in beliefs
Schwartenbeck, P., FitzGerald, T. H. B., & Dolan, R. J. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.10.067 · Cited by 82 · see citing works
“This quantity has previously been used successfully to model visual search behaviour (Itti and Baldi, 2009) and exploration (Sun et al., 2011).”
MethodRandom curiosity-driven exploration in deep reinforcement learning
Li, J., Shi, X., Li, J., Zhang, X., & Wang, J.-Z. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.08.024 · Cited by 68 · see citing works
All Thinking is ‘Wishful’ Thinking
Kruglanski, A. W., Jaśko, K., & Friston, K. J. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.03.004 · Cited by 76 · see citing works
UCB Exploration via Q-EnsemblesPreprint
Chen, R. Y., Sidor, S., Abbeel, P., & Schulman, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.01502 · Cited by 74 · see citing works
“[22], we define the information gain from observing an additional transition τn as Hτt|τ1,.”
BackgroundBalancing new against old information: the role of puzzlement surprise in learning
Faraji, M., Preuschoff, K., & Gerstner, W. (2018) · Neural Computation · doi:10.1162/neco_a_01025 · Cited by 68 · see citing works
“The maze exploration task is similar to tasks used in behavioral neuroscience and robotics (Morris, Garrard, Rawlins, & O’Keefe, 1982; Gillner & Mallot, 1998; Nelson, Grant, Galeotti, & Rhody, 2004; Sun et al., 2011; Rezende & Gerstner, 2014).”
BackgroundMethodInfluentialTschantz, A., Baltieri, M., Seth, A. K., & Buckley, C. L. (2020) · Sussex Research Online (University of Sussex) · Cited by 69 · see citing works
Reinforcement Learning through Active InferencePreprint
Tschantz, A., Millidge, B., Seth, A. K., & Buckley, C. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12636 · Cited by 72 · see citing works
“Information gain additionally has a substantial history outside the RL framework, going back to (Lindley, 1956; Still & Precup, 2012; Sun et al., 2011).”
BackgroundA Multi-scale View of the Emergent Complexity of Life: A Free-Energy Proposal
Hesp, C., Ramstead, M., Constant, A., Badcock, P., Kirchhoff, M., & Friston, K. (2019) · Springer Proceedings in Complexity · doi:10.1007/978-3-030-00075-2_7 · Cited by 63 · see citing works
Is Pessimism Provably Efficient for Offline RL?Preprint
Jin, Y., Yang, Z.-R., & Wang, Z.-R. (2020) · International Conference on Machine Learning · doi:10.48550/arxiv.2012.15085 · Cited by 454 · see citing works
“In particular, such a mutual information can be cast as the information gain (Schmidhuber, 1991; 2010; Sun et al., 2011; Still and Precup, 2012; Houthooft et al., 2016; Russo and Van Roy, 2016; 2018) for estimating w h , which is induced by observing � ( s h , a h ) in addition to D .”
BackgroundDegeneracy and Redundancy in Active Inference
Sajid, N., Parr, T., Hope, T. M., Price, C. J., & Friston, K. J. (2020) · Cerebral Cortex · doi:10.1093/cercor/bhaa148 · Cited by 49 · see citing works
The strategic student approach for life-long exploration and learning
Lopes, M. C., & Oudeyer, P. (2012) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2012.6400807 · Cited by 56 · see citing works
“For instance by learning how to learn learning strategies [14] or by searching for the simplest still unexplained problem [58].”
BackgroundEstimating Risk and Uncertainty in Deep Reinforcement LearningPreprint
Clements, W. R., van Delft, B., Robaglia, B.-M., Slaoui, R. B., & Toth, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09638 · Cited by 54 · see citing works
An Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey
Aubret, A., Matignon, L., & Hassas, S. (2023) · Entropy · doi:10.3390/e25020327 · Cited by 52 · see citing works
“The expected information gain [Little and Sommer 2013; Sun et al. 2011] over a forward or density model parameterized by φ can be formulated as:”
BackgroundInfluentialNROWAN-DQN: A stable noisy network with noise reduction and online weight adjustment for exploration
Han, S. D., Zhou, W., Lu, J., Liu, J., & Lü, S. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117343 · Cited by 42 · see citing works
“One kind of curiosity mechanism is generated based on the unknown information of the environment, such as Bayesian exploration in an unknown dynamic environment [14] and the theory based on the concept of maximizing intrinsic reward for active creation or discovery of new states [15].”
BackgroundCuriosity-Driven Experience Prioritization via Density Estimation
Zhao, R., & Tresp, V. (2019) · arXiv · doi:10.48550/arxiv.1902.08039 · Cited by 47 · see citing works
“Curiosity-driven exploration is a well-studied topic in reinforcement learning [33, 34, 43, 44, 48].”
BackgroundUniversal Knowledge-Seeking Agents for Stochastic Environments
Orseau, L., Lattimore, T., & Hütter, M. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40935-6_12 · Cited by 34 · see citing works
Reward maximization through discrete active inference
Da Costa, L., Sajid, N., Parr, T., Friston, K., & Smith, R. (2023) · Neural Computation · doi:10.1162/neco_a_01574 · Cited by 35 · see citing works
“…in machine learning and robotics (Barto et al., 2013; Deci & Ryan, 1985; Oudeyer & Kaplan, 2007) and expected Bayesian surprise in visual search (Itti & Baldi, 2009; Sun et al., 2011) and the organization of our visual apparatus (Barlow, 1961, 1974; Linsker, 1990; Optican & Richmond, 1987).”
BackgroundAction and Perception as Divergence MinimizationPreprint
Hafner, D., Ortega, P. A., Ba, J. L., Parr, T., Friston, K. J., & Heess, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.01791 · Cited by 32 · see citing works
“…(Lindley et al., 1956; MacKay, 1992b) that encourages inputs that are expected to convey the most bits about the representation to maximally reduce uncertainty in the belief, Information gain can be estimated by planning (Sun et al., 2011) or from past environment interaction (Schmidhuber, 1991).”
BackgroundAutomated Anomaly Detection via Curiosity-Guided Search and Self-Imitation Learning
Li, Y., Chen, Z., Zha, D., Zhou, K., Jin, H., Chen, H., & Hu, X. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3105636 · Cited by 44 · see citing works
Learning action-oriented models through active inferencePreprint
Tschantz, A., Seth, A. K., & Buckley, C. L. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/764969 · Cited by 28 · see citing works
Artificial curiosity with planning for autonomous perceptual and cognitive development
Luciw, M., Graziano, V., Ring, M. B., & Schmidhuber, J. (2011) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2011.6037356 · Cited by 27 · see citing works
“Another potentially serious over-simplification results when the RL task is reduced to maximizing the expected intrinsic reward at the next time step, rather than using the expected future discounted reward [19].”
BackgroundGoekoop, R., & de Kleijn, R. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.12.021 · Cited by 27 · see citing works
Automated Gameplay Testing and Validation With Curiosity-Conditioned Proximal Trajectories
Sestini, A., Gisslén, L., Bergdahl, J., Tollmar, K., & Bagdanov, A. D. (2022) · IEEE Transactions on Games · doi:10.1109/tg.2022.3226910 · Cited by 33 · see citing works
“Stadie et al. [34] proposed to assign exploration bonuses from a concurrently learned model of the system dynamics, while Sun et al. [35] showed that the optimal exploration strategy can be effectively approximated by solving a sequence of dynamic programming problems.”
BackgroundControl What You Can: Intrinsically Motivated Task-Planning AgentPreprint
Blaes, S., Vlastelica, M., Zhu, J.-J., & Martius, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.08190 · Cited by 26 · see citing works
Hierarchical reinforcement learning as creative problem solving
Colin, T. R., Belpaeme, T., Cangelosi, A., & Hemion, N. (2016) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2016.08.021 · Cited by 23 · see citing works
“Some of that work investigates optimal exploration from a Bayesian perspective [64], although that remains unfeasible for the general RL problem [65, pp. 25–28].”
Ready Policy One: World Building Through Active LearningPreprint
Ball, P., Parker-Holder, J., Pacchiano, A., Choromański, K., & Roberts, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02693 · Cited by 28 · see citing works
“Such issues have been highlighted previously (Schmidhuber, 1991; Sun et al., 2011), and motivate the design of our algorithm.”
BackgroundResolving uncertainty on the fly: modeling adaptive driving behavior as active inference
Engström, J., Wei, R., McDonald, A. D., García, A., O’Kelly, M., & Johnson, L. M. (2024) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2024.1341750 · Cited by 24 · see citing works
Lesort, T., Lomonaco, V., Stoian, A., Maltoni, D., Filliat, D., & Díaz-Rodríguez, N. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.12.004 · Cited by 531 · see citing works
“This idea is related to online change detection algorithms (Sarkar and Meeker, 1998; Moens and Zenon, 2018) or Bayesian surprise (Sun et al., 2011) in ML.”
MethodFields, C., Fabrocini, F., Friston, K. J., Glazebrook, J. F., Hazan, H., Levin, M. G., & Marcianò, A. (2023) · IEEE Transactions on Molecular Biological and Multi-Scale Communications · doi:10.1109/tmbmc.2023.3272158 · Cited by 30 · see citing works
Pezzulo, G., D’Amato, L., Mannella, F., Priorelli, M., de Maele, T. V., Stoianov, I. P., & Friston, K. J. (2024) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15118 · Cited by 23 · see citing works
Curiosity-Driven Reinforcement Learning with Homeostatic Regulation
Magrans, I., & Ryota Kanai, C. (2018) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489075 · Cited by 22 · see citing works
“It is interesting to note that [18] also proposed a recursive expression to describe the expected information gain of a an agent interacting with a more generic non-Markovian environment.”
BackgroundCultivating creativity: predictive brains and the enlightened room problem
Constant, A., Friston, K. J., & Clark, A. (2024) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2022.0415 · Cited by 19 · see citing works
Depeweg, S., Hernández-Lobato, J. M., Doshi‐Velez, F., & Udluft, S. (2017) · arXiv (Cornell University) · Cited by 19 · see citing works
A Framework for Reinforcement Learning and Planning.Preprint
Moerland, T. M., Broekens, J., & Jonker, C. M. (2020) · arXiv (Cornell University) · Cited by 20 · see citing works
Resilience and active inference
Miller, M. M., Albarracin, M., Pitliya, R. J., Kiefer, A. B., Mago, J., Gorman, C., Friston, K. J., & Ramstead, M. J. D. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.1059117 · Cited by 18 · see citing works
“, inference) and selforganization; ranging from Jaynes’ maximum entropy principle, through to universal computation via the minimization of algorithmic complexity and description lengths (Wallace and Dowe, 1999; Hutter, 2004; Sun et al., 2011; Ramstead et al., 2022).”
MethodLoued‐Khenissi, L., & Preuschoff, K. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.00005 · Cited by 19 · see citing works
“The implication of information collection is not trivial: it supports the notion that an agent may want to maximize her entropy (Schwartenbeck et al., 2013) and in so doing “seek” surprise (Clark, 2018), or a state of expecting the unexpected (Sun et al., 2011.”
BackgroundCosta, L. D., Sajid, N., Parr, T., Friston, K. J., & Smith, R. S. (2020) · arXiv (Cornell University) · Cited by 19 · see citing works
Li, Z., Chen, W., & Yang, J. (2023) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3221907 · Cited by 19 · see citing works
“Therefore, the reward function, which the agent targets to maximise while making its next movement, is defined according to the information gain using some informative measures [19], for example, entropy, Kullback-Leibler divergence and variance.”
Modern Artificial Neural Networks: Is Evolution Cleverer?
Bahmer, A., Gupta, D. S., & Effenberger, F. (2023) · Neural Computation · doi:10.1162/neco_a_01575 · Cited by 16 · see citing works
“Dimension reduction is closely related to minimizing statistical complexity (used in variational autoencoders) and compressibility and efficiency in the sense of Kolmogorov complexity and ensuing formulations of universal computation (Hutter, 2006; Wallace & Dowe, 1999; Sun et al., 2011).”
BackgroundSMiRL: Surprise Minimizing RL in Dynamic EnvironmentsPreprint
Berseth, G., Geng, D., Devin, C., Finn, C., Jayaraman, D., & Levine, S. (2019) · arXiv.org · doi:10.48550/arxiv.1912.05510 · Cited by 22 · see citing works
“, 2016), via surprise maximization (Achiam & Sastry, 2017; Schmidhuber, 1991; Sun et al., 2011), and through other novelty-based reward bonuses (Lehman & Stanley, 2011; Burda et al.”
BackgroundGeometric methods for sampling, optimization, inference, and adaptive agents
Barp, A., Costa, L. D., França, G., Friston, K. J., Girolami, M., Jordan, M. I., & Pavliotis, G. A. (2022) · Handbook of statistics · doi:10.1016/bs.host.2022.03.005 · Cited by 15 · see citing works
Explore and Explain: Self-supervised Navigation and Recounting
Bigazzi, R., Landi, F., Cornia, M., Cascianelli, S., Baraldi, L., & Cucchiara, R. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412628 · Cited by 15 · see citing works
“[20] proposed to use information gain and compression as intrinsic rewards, while Klyubin et al.”
SMiRL: Surprise Minimizing RL in Dynamic EnvironmentsPreprint
Berseth, G., Geng, D., Devin, C., Finn, C., Jayaraman, D., & Levine, S. (2019) · arXiv (Cornell University) · Cited by 15 · see citing works
MetaCURE: Meta Reinforcement Learning with Empowerment-Driven ExplorationPreprint
Zhang, J., Wang, J., Hu, H., Chen, T., Chen, Y., Fan, C., & Zhang, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08170 · Cited by 15 · see citing works
“Sun et al. (2011) discusses exploration with information gain, but is restricted to planning problems and requires an oracle estimating the posterior.”
Active Inference, Bayesian Optimal Design, and Expected Utility
Sajid, N., Costa, L. D., Parr, T., & Friston, K. J. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009026949.007 · Cited by 14 · see citing works
Confidence-based progress-driven self-generated goals for skill acquisition in developmental robots
Ngo, H., Luciw, M., Förster, A., & Schmidhuber, J. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00833 · Cited by 13 · see citing works
“Yi et al. (2011) develop a theoretically optimal framework based on the Bayesian meth-ods, in which the agent aims to maximize the information gain in estimating the distribution of model parameters.”
MethodLiakoni, V., Lehmann, M. P., Modirshanechi, A., Brea, J., Lutti, A., Gerstner, W., & Preuschoff, K. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118780 · Cited by 14 · see citing works
AutoOD: Automated Outlier Detection via Curiosity-guided Search and Self-imitation LearningPreprint
Li, Y., Chen, Z., Zha, D., Zhou, K., Jin, H., Chen, H., & Hu, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.11321 · Cited by 13 · see citing works
“To address these problems, AutoOD builds on the theory of curiosity-driven exploration [30], aiming to encourage the controller to seek out regions in the searching spaces that are relatively unexplored.”
BackgroundTschantz, A., Baltieri, M., Seth, A. K., & Buckley, C. L. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207382 · Cited by 74 · see citing works
“Reference [67] developed a Bayesian framework in order to maximize information gain via dynamic programming, however, the experiments were limited to discrete state spaces using tabular MDPs.”
MethodBrändle, F., Binz, M., & Schulz, E. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009026949.008 · Cited by 12 · see citing works
Horváth, L., Colcombe, S. J., Milham, M. P., Ray, S., Schwartenbeck, P., & Ostwald, D. (2021) · Computational Brain & Behavior · doi:10.1007/s42113-021-00112-3 · Cited by 12 · see citing works
“Notably, we adopt a Bayesian treatment of exploration and quantify the information bonus as the expected Bayesian surprise (Itti & Baldi, 2009; Sun et al., 2011; Ostwald et al., 2012).”
MethodSun, X., & Bischl, B. (2019) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci44817.2019.9003114 · Cited by 11 · see citing works
“Upon additional observation of action at and state st+1 pair on top of trajectory history ξt, the posterior on the distribution of the environmental parameter θ, p(θ|ξt), could be updated to be p(θ|ξt+1) in a Bayesian way as derived in Equation (43), which is first proposed in [28].”
BackgroundLearning-Driven Exploration for Reinforcement Learning
Usama, M., & Chang, D. E. (2021) · International Conference on Control, Automation and Systems (ICCAS) · doi:10.23919/iccas52745.2021.9649810 · Cited by 9 · see citing works
“strategy based on maximization of information gain about the agent’s belief of the environment dynamics. Using information gain for exploration can be traced to [12] and has been further explored in [13,10,14]. Practical reinforcement learning algorithms often utilize simple exploration heuristics, such as -greedy and…”
Bayesian Brains and the Rényi Divergence
Sajid, N., Faccio, F., Costa, L. D., Parr, T., Schmidhuber, J., & Friston, K. J. (2022) · Neural Computation · doi:10.1162/neco_a_01484 · Cited by 9 · see citing works
Ghazi-Zahedi, K., Martius, G., & Ay, N. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00801 · Cited by 10 · see citing works
New Millennium AI and the Convergence of History: Update of 2012
Schmidhuber, J. (2012) · The frontiers collection · doi:10.1007/978-3-642-32560-1_4 · Cited by 12 · see citing works
“Recent work has demonstrated exploration that is both optimal and feasible [106] for limited scenarios, but much remains to be done for challenging, high-dimensional, partially observable worlds.”
BackgroundContinual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay ProcessesPreprint
Lesort, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00487 · Cited by 14 · see citing works
“When a robot needs to learn a task in a known or unknown environment, it must collect its own training data in the real world Wong (2016). Data serves as the basis for environment exploration and comprehension.”
BackgroundMethodCao, F., & Guo, X. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108663 · Cited by 8 · see citing works
Learning Intrinsic Rewards as a Bi-Level Optimization Problem
Stadie, B. C., Zhang, L., & Ba, J. L. (2020) · Uncertainty in Artificial Intelligence · Cited by 8 · see citing works
Adaptive Procedural Task Generation for Hard-Exploration ProblemsPreprint
Fang, K., Zhu, Y., Savarese, S., & Fei-Fei, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00350 · Cited by 10 · see citing works
“Another family of exploration techniques is derived from an informationtheoretical perspective as maximizing information gain of actions (Houthooft et al., 2016; Sun et al., 2011).”
BackgroundEvaluating Agents without RewardsPreprint
Matusch, B., Ba, J. L., & Hafner, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11538 · Cited by 14 · see citing works
“Information gain has led to successful exploration in RL (Sun et al., 2011; Houthooft et al., 2016; Shyam et al., 2018; Mirchev et al., 2018; Sekar et al., 2020).”
BackgroundMethodVASE: Variational Assorted Surprise Exploration for Reinforcement Learning
Xu, H., Szymanski, L., & Brendan Mccane, A. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3105140 · Cited by 7 · see citing works
“The main idea is that agents should take actions to maximize the sum of reductions in entropy [19] ∑”
BackgroundExploring Parameter Space with Structured Noise for Meta-Reinforcement Learning
Xu, H., Zhang, C., Wang, J., Ouyang, D., Zheng, Y., & Shao, J. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/436 · Cited by 7 · see citing works
“The concept of curiosity [Sun et al. , 2011] is beneficial to the exploration.”
BackgroundUnburdening the Shoulders of Giants: A Quest for Disconnected Academic Psychology
Krpan, D. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620904775 · Cited by 7 · see citing works
Active Learning for Autonomous Intelligent Agents: Exploration, Curiosity, and InteractionPreprint
Lopes, M. C., & Montesano, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1403.1497 · Cited by 8 · see citing works
“As a generalization of exploration methods in reinforcement learning, such as (Brafman and Tennenholtz, 2003), ideas have been suggested such as plan- ning to be surprised (Sun et al., 2011) or the combination of empirical learning progress with visit counts (Hester and Stone, 2012).”
MethodSchultheis, M., Belousov, B., Abdulsamad, H., & Peters, J. (2019) · Technischen Universität Darmstadt · doi:10.26083/tuprints-00020578 · Cited by 6 · see citing works
An Active Information Seeking Model for Goal-oriented Vision-and-Language Tasks.Preprint
Abbasnejad, E., Wu, Q., Abbasnejad, I., Shi, Q., & van den Hengel, A. (2018) · arXiv (Cornell University) · Cited by 9 · see citing works
Learn to Effectively Explore in Context-Based Meta-RL.Preprint
Zhang, J., Wang, J., Hu, H., Chen, Y., Fan, C., & Zhang, C. (2020) · arXiv (Cornell University) · Cited by 6 · see citing works
Information-based learning by agents in unbounded state spaces
Mobin, S., Arnemann, J., & Sommer, F. (2014) · neural information processing systems · Cited by 5 · see citing works
Computing Machinery, Surprise and Originality
Delacroix, S. (2020) · Philosophy & Technology · doi:10.1007/s13347-021-00453-8 · Cited by 4 · see citing works
“13 Referred to in Taylor (2006). 14 ‘Originality, far from being autonomous, is contingent at every point upon the unoriginal structures that inform it’ (Turner, 1991, p.”
BackgroundImplicit Generative Modeling for Efficient ExplorationPreprint
Ratzlaff, N., Bai, Q., Li, F., & Xu, W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08017 · Cited by 4 · see citing works
“Schmidhuber (1991) and Sun et al. (2011) proposed using the learning progress of the agent as an intrinsic reward.”
Control flow in active inference systemsPreprint
Fields, C., Fabrocini, F., Friston, K. J., Glazebrook, J. F., Hazan, H., Levin, M. G., & Marcianò, A. (2023) · doi:10.31219/osf.io/8e4ra · Cited by 4 · see citing works
Balancing New Against Old Information: The Role of SurprisePreprint
Faraji, M. J., Preuschoff, K., & Gerstner, W. (2016) · arXiv (Cornell University) · Cited by 3 · see citing works
A Bandit Framework for Optimal Selection of Reinforcement Learning AgentsPreprint
Merentitis, A., Rasul, K., Vollgraf, R., Sheikh, A.-S., & Bergmann, U. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.03657 · Cited by 3 · see citing works
“so very beneficial, both with respect to focusing early onto the most promising agents, as well as on being able to select the best agent after a relatively small number of steps. It has been shown in [13] that it is beneficial for agents to take actions that maximize the reduction in uncertainty about the environment d…”
BackgroundTowards Informed Exploration for Deep Reinforcement Learning
Tang, H. (2019) · eScholarship (California Digital Library) · Cited by 3 · see citing works
Applications of the Free Energy Principle to Machine Learning and NeurosciencePreprint
Millidge, B. (2021) · arXiv (Cornell University) · doi:10.7488/era/1501 · Cited by 7 · see citing works
“Similarly, our approach allows us to understand and rationalize a number of approaches in the literature (Klyubin et al., 2005; Oudeyer & Kaplan, 2009; Sun et al., 2011) which add additional exploratory terms to their reward-maximizing agents, with the heuristic justification that they should increase exploration.”
Method
- Authors
- Sun, Y., Gomez, F., & Schmidhuber, J.
- Affiliations
- Dalle Molle Institute for Artificial Intelligence Research (CH)
- Venue
- Lecture notes in computer science
- Publisher
- Springer Berlin Heidelberg
- Published
- 2011
- Pages
- 41-51
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W1550989509
- Cited by
- 134
- References
- 11
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 5 · 2023: 11 · 2022: 12 · 2021: 17 · 2020: 30 · 2019: 17 · 2018: 7 · 2017: 6 · 2016: 9 · 2015: 2 · 2014: 3 · 2013: 5 · 2012: 3
- FWCI
- 3.25
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Lecture notes in computer science (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- dynamic environments, artificial general intelligence, surprise, exploration planning, unknown environments
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
