The DOI matches its Crossref record; 10 of the 11 references it cites were found in a registry or an index.
Planning to Be Surprised: Optimal Bayesian Exploration in Dynamic Environments
Open accessOverview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This underwrites Bayesian experimental design [134] and active learning in statistics [135], intrinsic motivation and artificial curiosity in machine learning and robotics [136, 137, 138, 139, 140].”
From The free energy principle made simpler but not too simple (2023) · cites this paper as [139]
In the citing paper
“Adaptive sampling methods are also known in reinforcement learning that can achieve optimal Bayesian exploration of Markov Decision Process environments (Sun et al., 2011; Guez et al., 2012).”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Sun et al., 2011
Citations
Cites
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
Fedorov, V. V., Studden, W. J., & Klimko, E. M. (1972) · Cited by 2,057
Theory of Optimal Experiments.
Herzberg, A. M., Wynn, H. P., Fedorov, V. V., Studden, W. J., & Klimko, E. M. (1972) · Biometrika · doi:10.2307/2334826 · Cited by 1,896 · cited 2×
“In particular, measuring learning progress using KL divergence dates back to the 50’s [2,4].”
BackgroundBayesian Experimental Design: A Review
Chaloner, K. M., & Verdinelli, I. (1995) · Statistical Science · doi:10.1214/ss/1177009939 · Cited by 1,843 · cited 1×
“The idea of actively selecting queries to accelerate learning process has a long history [1, 2, 7], and has received a lot of attention in recent decades, primarily in the context of active learning [8] and artificial curiosity [6].”
BackgroundOn a measure of the information provided by an experiment
Lindley, D. V. (1956) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177728069 · Cited by 1,509 · cited 1×
“In particular, measuring learning progress using KL divergence dates back to the 50’s [2,4].”
BackgroundFormal theory of creativity, fun, and intrinsic motivation (1990–2010)
Schmidhuber, J. (2010) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2010.2056368 · Cited by 747 · cited 2×
“Conceptually, however, this work is closely connected to artificial curiosity and intrinsically motivated reinforcement learning [6,7,9] for agents that actively explore the environment without an external reward signal.”
BackgroundIntrinsically Motivated Reinforcement Learning
Singh, S. P., Barto, A. G., & Chentanez, N. (2005) · doi:10.21236/ada440280 · Cited by 614 · cited 1×
“Conceptually, however, this work is closely connected to artificial curiosity and intrinsically motivated reinforcement learning [6,7,9] for agents that actively explore the environment without an external reward signal.”
BackgroundCurious model-building control systems
Schmidhuber, J. (1991) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1991.170605 · Cited by 601 · cited 2×
“Conceptually, however, this work is closely connected to artificial curiosity and intrinsically motivated reinforcement learning [6,7,9] for agents that actively explore the environment without an external reward signal.”
BackgroundBayesian Surprise Attracts Human Attention
Itti, L., & Baldi, P. (2005) · neural information processing systems · Cited by 363
Planning to Be Surprised: Optimal Bayesian Exploration in Dynamic EnvironmentsPreprint
Sun, Y., Gomez, F. J., & Schmidhuber, J. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.5708 · Cited by 25
Self-citation
Cited by
Sorted by citations
VIME: Variational Information Maximizing ExplorationPreprint
Houthooft, R., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09674 · Cited by 597
#Exploration: A Study of Count-Based Exploration for Deep Reinforcement LearningPreprint
Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04717 · Cited by 504
“Bayesian RL methods (Kolter & Ng, 2009; Guez et al., 2014; Sun et al., 2011; Ghavamzadeh et al., 2015), which keep track of a distribution over MDPs, are an alternative to optimism-based methods.”
MethodWulfram Gerstner, B., Lehmann, M., Liakoni, V., Corneil, D., & Brea, J. (2018) · Frontiers in Neural Circuits · doi:10.3389/fncir.2018.00053 · Cited by 354
“Theoreticians have studied these ideas in the context of curiosity (Schmidhuber, 2010), information gain during active exploration (Storck et al., 1995; Schmidhuber, 2006; Sun et al., 2011; Little and Sommer, 2013; Friston et al., 2016), and via formal definitions of surprise (Shannon, 1948; Storck et al.”
BackgroundComputational mechanisms of curiosity and goal-directed exploration
Schwartenbeck, P., Passecker, J., Hauser, T. U., FitzGerald, T. H. B., Kronbichler, M., & Friston, K. J. (2019) · eLife · doi:10.7554/elife.41703 · Cited by 278
Planning and navigation as active inference
Kaplan, R., & Friston, K. J. (2018) · Biological Cybernetics · doi:10.1007/s00422-018-0753-2 · Cited by 203
Active inference on discrete state-spaces: A synthesis
Costa, L. D., Parr, T., Sajid, N., Veselic, S., Neacsu, V., & Friston, K. J. (2020) · UCL Discovery (University College London) · Cited by 204
Intrinsic Valuation of Information in Decision Making under Uncertainty
Bennett, D., Bode, S., Brydevall, M., Warren, H. A., & Murawski, C. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005020 · Cited by 198
“In particular, it is possible to reparametrise Eq 6 such that intrinsic valuation of information is expressed in terms of value bonus for information, rather than an aversion to states’ uncertainty (see also Sun, Gomez & Schmidhuber, 2011 [49] for a discussion of conceptual differences between information reward signa…”
BackgroundSentience and the Origins of Consciousness: From Cartesian Duality to Markovian Monism
Friston, K. J., Wiese, W., & Hobson, J. A. (2020) · Entropy · doi:10.3390/e22050516 · Cited by 186
“, expected surprisal or uncertainty), we now have a plausible description of intentional behaviour that will, to all intents and purposes, look like uncertainty resolving, information seeking, epistemic foraging [81,83,84,26,85,86,87,88,89,90].”
BackgroundSelf-Supervised Exploration via DisagreementPreprint
Pathak, D., Gandhi, D., & Gupta, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04161 · Cited by 190
“This prediction error can also be formulated as surprise (Achiam & Sastry, 2017; Schmidhuber, 1991a; Sun et al., 2011).”
BackgroundThe free energy principle made simpler but not too simple
Friston, K., Da Costa, L., Sajid, N., Heins, C., Ueltzhöffer, K., Pavliotis, G. A., & Parr, T. (2023) · Physics Reports · doi:10.1016/j.physrep.2023.07.001 · Cited by 163
“This underwrites Bayesian experimental design [134] and active learning in statistics [135], intrinsic motivation and artificial curiosity in machine learning and robotics [136, 137, 138, 139, 140].”
Planning to Explore via Self-Supervised World Models
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., & Pathak, D. (2020) · arXiv · doi:10.48550/arxiv.2005.05960 · Cited by 129
“Sun et al. (2011) derived a model-based reinforcement learning agent that can optimize the infinite-horizon information gain and experimented with it in tabular settings.”
BackgroundLearning action-oriented models through active inference
Tschantz, A., Seth, A. K., & Buckley, C. L. (2020) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007805 · Cited by 121
“…models to determine epistemic (or intrinsic, information-seeking, uncertainty reducing ) actions that attempt to minimize the model uncertainty in a directed manner (Stadie et al., 2015; Houthooft et al., 2016; Sun et al., 2011; Friston et al., 2015; Burda et al., 2018; Friston et al., 2017).”
BackgroundToward an integration of deep learning and neuroscience
Marblestone, A. H., Wayne, G., & Kording, K. P. (2016) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2016.00094 · Cited by 119
“Adaptive sampling methods are also known in reinforcement learning that can achieve optimal Bayesian exploration of Markov Decision Process environments (Sun et al., 2011; Guez et al., 2012).”
MethodFriston, K. J., Costa, L. D., Hafner, D., Hesp, C., & Parr, T. (2021) · Neural Computation · doi:10.1162/neco_a_01351 · Cited by 150
“This is mathematically the same as expected Bayesian surprise and mutual information that underwrites salience in visual search (Itti and Baldi, 2009; Sun et al., 2011) and the organisation of our visual apparatus (Barlow, 1961; Barlow, 1974; Linsker, 1990; Optican and Richmond, 1987).”
BackgroundSchmidhuber, J. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00313 · Cited by 135
“xp loration seem obvious. One should analyze theoretically and experimentally under which conditions the creation of self-generated tasks can accelerate the solution to externally generated tasks—see [30, 54, 37, 38, 19, 4, 28, 62] for previous simple experimental studies in this vein. 6.1 Self-Reference Through Novel…”
BackgroundResultInfluentialPath integrals, particular kinds, and strange things
Friston, K. J., Costa, L. D., Sakthivadivel, D. A. R., Heins, C., Pavliotis, G. A., Ramstead, M. J. D., & Parr, T. (2023) · Physics of Life Reviews · doi:10.1016/j.plrev.2023.08.016 · Cited by 145
Modern Bayesian Experimental Design
Rainforth, T., Foster, A., Ivanova, D. R., & Smith, F. B. (2024) · Statistical Science · doi:10.1214/23-sts915 · Cited by 192
Evidence for surprise minimization over value maximization in choice behavior
Schwartenbeck, P., FitzGerald, T. H. B., Mathys, C., Dolan, R., Kronbichler, M., & Friston, K. (2015) · Scientific Reports · doi:10.1038/srep16575 · Cited by 96
An Exploration of Embodied Visual Exploration
Ramakrishnan, S. K., Jayaraman, D., & Grauman, K. L. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01437-z · Cited by 105
“Reconstruction-based exploration approaches also relate to classical informationgain based exploration, where the goal is to explore in order to reduce the uncertainty in the agent’s model of the world [13,63,66].”
Representation Wars: Enacting an Armistice Through Active Inference
Constant, A., Clark, A., & Friston, K. J. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.598733 · Cited by 102
“In visual neurosciences, this is called salience and is a key determinant of epistemic foraging or exploratory behavior (Itti and Baldi, 2009; Sun et al., 2011).”
BackgroundFriston, K. J., Sajid, N., Quiroga‐Martinez, D. R., Parr, T., Price, C. J., & Holmes, E. (2020) · Hearing Research · doi:10.1016/j.heares.2020.107998 · Cited by 113
Whence the Expected Free Energy?
Millidge, B., Tschantz, A., & Buckley, C. L. (2021) · Neural Computation · doi:10.1162/neco_a_01354 · Cited by 114
World model learning and inference
Friston, K. J., Moran, R., Nagai, Y., Taniguchi, T., Gomi, H., & Tenenbaum, J. B. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.09.011 · Cited by 99
“…both pragmatic (extrinsic) and epistemic (intrinsic) imperatives for good plans that resolve the exploration–exploitation dilemma (Cohen, McClure, & Yu, 2007; Friston et al., 2015; Schmidhuber, 2006; Still & Precup, 2012; Sun, Gomez, & Schmidhuber, 2011; Tschantz, Baltieri, Seth, & Buckley, 2020).”
Play in predictive minds: A cognitive theory of play.
Andersen, M. M., Kiverstein, J., Miller, M., & Roepstorff, A. (2023) · Psychological Review · doi:10.1037/rev0000369 · Cited by 122
Neural signals encoding shifts in beliefs
Schwartenbeck, P., FitzGerald, T. H. B., & Dolan, R. J. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.10.067 · Cited by 84
“This quantity has previously been used successfully to model visual search behaviour (Itti and Baldi, 2009) and exploration (Sun et al., 2011).”
MethodRandom curiosity-driven exploration in deep reinforcement learning
Li, J., Shi, X., Li, J., Zhang, X., & Wang, J.-Z. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.08.024 · Cited by 68
All Thinking is ‘Wishful’ Thinking
Kruglanski, A. W., Jaśko, K., & Friston, K. J. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.03.004 · Cited by 76
UCB Exploration via Q-EnsemblesPreprint
Chen, R. Y., Sidor, S., Abbeel, P., & Schulman, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.01502 · Cited by 152
“[22], we define the information gain from observing an additional transition τn as Hτt|τ1,.”
BackgroundSome Considerations on Learning to Explore via Meta-Reinforcement Learning
Stadie, B. C., Yang, G., Houthooft, R., Chen, X., Duan, Y., Wu, Y., Abbeel, P., & Sutskever, I. (2018) · arXiv · doi:10.48550/arxiv.1803.01118 · Cited by 74
Tschantz, A., Baltieri, M., Seth, A. K., & Buckley, C. L. (2020) · Sussex Research Online (University of Sussex) · Cited by 69
Balancing new against old information: the role of puzzlement surprise in learning
Faraji, M., Preuschoff, K., & Gerstner, W. (2018) · Neural Computation · doi:10.1162/neco_a_01025 · Cited by 68
“The maze exploration task is similar to tasks used in behavioral neuroscience and robotics (Morris, Garrard, Rawlins, & O’Keefe, 1982; Gillner & Mallot, 1998; Nelson, Grant, Galeotti, & Rhody, 2004; Sun et al., 2011; Rezende & Gerstner, 2014).”
BackgroundMethodInfluentialReinforcement Learning through Active InferencePreprint
Tschantz, A., Millidge, B., Seth, A. K., & Buckley, C. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12636 · Cited by 96
“Information gain additionally has a substantial history outside the RL framework, going back to (Lindley, 1956; Still & Precup, 2012; Sun et al., 2011).”
BackgroundA Multi-scale View of the Emergent Complexity of Life: A Free-Energy Proposal
Hesp, C., Ramstead, M., Constant, A., Badcock, P., Kirchhoff, M., & Friston, K. (2019) · Springer Proceedings in Complexity · doi:10.1007/978-3-030-00075-2_7 · Cited by 63
Is Pessimism Provably Efficient for Offline RL?Preprint
Jin, Y., Yang, Z.-R., & Wang, Z.-R. (2020) · International Conference on Machine Learning · doi:10.48550/arxiv.2012.15085 · Cited by 454
“In particular, such a mutual information can be cast as the information gain (Schmidhuber, 1991; 2010; Sun et al., 2011; Still and Precup, 2012; Houthooft et al., 2016; Russo and Van Roy, 2016; 2018) for estimating w h , which is induced by observing � ( s h , a h ) in addition to D .”
BackgroundDegeneracy and Redundancy in Active Inference
Sajid, N., Parr, T., Hope, T. M., Price, C. J., & Friston, K. J. (2020) · Cerebral Cortex · doi:10.1093/cercor/bhaa148 · Cited by 49
The strategic student approach for life-long exploration and learning
Lopes, M. C., & Oudeyer, P. (2012) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2012.6400807 · Cited by 56
“For instance by learning how to learn learning strategies [14] or by searching for the simplest still unexplained problem [58].”
BackgroundEstimating Risk and Uncertainty in Deep Reinforcement LearningPreprint
Clements, W. R., van Delft, B., Robaglia, B.-M., Slaoui, R. B., & Toth, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09638 · Cited by 54
An Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey
Aubret, A., Matignon, L., & Hassas, S. (2023) · Entropy · doi:10.3390/e25020327 · Cited by 52
“The expected information gain [Little and Sommer 2013; Sun et al. 2011] over a forward or density model parameterized by φ can be formulated as:”
BackgroundInfluentialNROWAN-DQN: A stable noisy network with noise reduction and online weight adjustment for exploration
Han, S. D., Zhou, W., Lu, J., Liu, J., & Lü, S. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117343 · Cited by 45
“One kind of curiosity mechanism is generated based on the unknown information of the environment, such as Bayesian exploration in an unknown dynamic environment [14] and the theory based on the concept of maximizing intrinsic reward for active creation or discovery of new states [15].”
BackgroundCuriosity-Driven Experience Prioritization via Density Estimation
Zhao, R., & Tresp, V. (2019) · arXiv · doi:10.48550/arxiv.1902.08039 · Cited by 47
“Curiosity-driven exploration is a well-studied topic in reinforcement learning [33, 34, 43, 44, 48].”
BackgroundReward maximization through discrete active inference
Da Costa, L., Sajid, N., Parr, T., Friston, K., & Smith, R. (2023) · Neural Computation · doi:10.1162/neco_a_01574 · Cited by 35
“…in machine learning and robotics (Barto et al., 2013; Deci & Ryan, 1985; Oudeyer & Kaplan, 2007) and expected Bayesian surprise in visual search (Itti & Baldi, 2009; Sun et al., 2011) and the organization of our visual apparatus (Barlow, 1961, 1974; Linsker, 1990; Optican & Richmond, 1987).”
BackgroundUniversal Knowledge-Seeking Agents for Stochastic Environments
Orseau, L., Lattimore, T., & Hütter, M. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40935-6_12 · Cited by 34
Action and Perception as Divergence MinimizationPreprint
Hafner, D., Ortega, P. A., Ba, J. L., Parr, T., Friston, K. J., & Heess, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.01791 · Cited by 52
“…(Lindley et al., 1956; MacKay, 1992b) that encourages inputs that are expected to convey the most bits about the representation to maximally reduce uncertainty in the belief, Information gain can be estimated by planning (Sun et al., 2011) or from past environment interaction (Schmidhuber, 1991).”
BackgroundAutomated Anomaly Detection via Curiosity-Guided Search and Self-Imitation Learning
Li, Y., Chen, Z., Zha, D., Zhou, K., Jin, H., Chen, H., & Hu, X. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3105636 · Cited by 47
Learning action-oriented models through active inferencePreprint
Tschantz, A., Seth, A. K., & Buckley, C. L. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/764969 · Cited by 28
Artificial curiosity with planning for autonomous perceptual and cognitive development
Luciw, M., Graziano, V., Ring, M. B., & Schmidhuber, J. (2011) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2011.6037356 · Cited by 27
“Another potentially serious over-simplification results when the RL task is reduced to maximizing the expected intrinsic reward at the next time step, rather than using the expected future discounted reward [19].”
BackgroundGoekoop, R., & de Kleijn, R. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.12.021 · Cited by 27
Automated Gameplay Testing and Validation With Curiosity-Conditioned Proximal Trajectories
Sestini, A., Gisslén, L., Bergdahl, J., Tollmar, K., & Bagdanov, A. D. (2022) · IEEE Transactions on Games · doi:10.1109/tg.2022.3226910 · Cited by 33
“Stadie et al. [34] proposed to assign exploration bonuses from a concurrently learned model of the system dynamics, while Sun et al. [35] showed that the optimal exploration strategy can be effectively approximated by solving a sequence of dynamic programming problems.”
BackgroundControl What You Can: Intrinsically Motivated Task-Planning AgentPreprint
Blaes, S., Vlastelica, M., Zhu, J.-J., & Martius, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.08190 · Cited by 26
Hierarchical reinforcement learning as creative problem solving
Colin, T. R., Belpaeme, T., Cangelosi, A., & Hemion, N. (2016) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2016.08.021 · Cited by 26
“Some of that work investigates optimal exploration from a Bayesian perspective [64], although that remains unfeasible for the general RL problem [65, pp. 25–28].”
Ready Policy One: World Building Through Active LearningPreprint
Ball, P., Parker-Holder, J., Pacchiano, A., Choromański, K., & Roberts, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02693 · Cited by 29
“Such issues have been highlighted previously (Schmidhuber, 1991; Sun et al., 2011), and motivate the design of our algorithm.”
BackgroundResolving uncertainty on the fly: modeling adaptive driving behavior as active inference
Engström, J., Wei, R., McDonald, A. D., García, A., O’Kelly, M., & Johnson, L. M. (2024) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2024.1341750 · Cited by 24
Lesort, T., Lomonaco, V., Stoian, A., Maltoni, D., Filliat, D., & Díaz-Rodríguez, N. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.12.004 · Cited by 593
“This idea is related to online change detection algorithms (Sarkar and Meeker, 1998; Moens and Zenon, 2018) or Bayesian surprise (Sun et al., 2011) in ML.”
MethodFields, C., Fabrocini, F., Friston, K. J., Glazebrook, J. F., Hazan, H., Levin, M. G., & Marcianò, A. (2023) · IEEE Transactions on Molecular Biological and Multi-Scale Communications · doi:10.1109/tmbmc.2023.3272158 · Cited by 31
Pezzulo, G., D’Amato, L., Mannella, F., Priorelli, M., de Maele, T. V., Stoianov, I. P., & Friston, K. J. (2024) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15118 · Cited by 25
Curiosity-Driven Reinforcement Learning with Homeostatic Regulation
Magrans, I., & Ryota Kanai, C. (2018) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489075 · Cited by 22
“It is interesting to note that [18] also proposed a recursive expression to describe the expected information gain of a an agent interacting with a more generic non-Markovian environment.”
BackgroundCultivating creativity: predictive brains and the enlightened room problem
Constant, A., Friston, K. J., & Clark, A. (2024) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2022.0415 · Cited by 19
Depeweg, S., Hernández-Lobato, J. M., Doshi‐Velez, F., & Udluft, S. (2017) · arXiv (Cornell University) · Cited by 19
A Framework for Reinforcement Learning and Planning.Preprint
Moerland, T. M., Broekens, J., & Jonker, C. M. (2020) · arXiv (Cornell University) · Cited by 20
Resilience and active inference
Miller, M. M., Albarracin, M., Pitliya, R. J., Kiefer, A. B., Mago, J., Gorman, C., Friston, K. J., & Ramstead, M. J. D. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.1059117 · Cited by 20
“, inference) and selforganization; ranging from Jaynes’ maximum entropy principle, through to universal computation via the minimization of algorithmic complexity and description lengths (Wallace and Dowe, 1999; Hutter, 2004; Sun et al., 2011; Ramstead et al., 2022).”
MethodLoued‐Khenissi, L., & Preuschoff, K. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.00005 · Cited by 19
“The implication of information collection is not trivial: it supports the notion that an agent may want to maximize her entropy (Schwartenbeck et al., 2013) and in so doing “seek” surprise (Clark, 2018), or a state of expecting the unexpected (Sun et al., 2011.”
BackgroundCosta, L. D., Sajid, N., Parr, T., Friston, K. J., & Smith, R. S. (2020) · arXiv (Cornell University) · Cited by 19
Li, Z., Chen, W., & Yang, J. (2023) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3221907 · Cited by 19
“Therefore, the reward function, which the agent targets to maximise while making its next movement, is defined according to the information gain using some informative measures [19], for example, entropy, Kullback-Leibler divergence and variance.”
Modern Artificial Neural Networks: Is Evolution Cleverer?
Bahmer, A., Gupta, D. S., & Effenberger, F. (2023) · Neural Computation · doi:10.1162/neco_a_01575 · Cited by 17
“Dimension reduction is closely related to minimizing statistical complexity (used in variational autoencoders) and compressibility and efficiency in the sense of Kolmogorov complexity and ensuing formulations of universal computation (Hutter, 2006; Wallace & Dowe, 1999; Sun et al., 2011).”
BackgroundSMiRL: Surprise Minimizing Reinforcement Learning in Unstable EnvironmentsPreprint
Berseth, G., Geng, D., Devin, C., Rhinehart, N., Finn, C., Jayaraman, D., & Levine, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.05510 · Cited by 41
“, 2016), via surprise maximization (Achiam & Sastry, 2017; Schmidhuber, 1991; Sun et al., 2011), and through other novelty-based reward bonuses (Lehman & Stanley, 2011; Burda et al.”
BackgroundGeometric methods for sampling, optimization, inference, and adaptive agents
Barp, A., Costa, L. D., França, G., Friston, K. J., Girolami, M., Jordan, M. I., & Pavliotis, G. A. (2022) · Handbook of statistics · doi:10.1016/bs.host.2022.03.005 · Cited by 24
Explore and Explain: Self-supervised Navigation and Recounting
Bigazzi, R., Landi, F., Cornia, M., Cascianelli, S., Baraldi, L., & Cucchiara, R. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412628 · Cited by 15
“[20] proposed to use information gain and compression as intrinsic rewards, while Klyubin et al.”
SMiRL: Surprise Minimizing RL in Dynamic EnvironmentsPreprint
Berseth, G., Geng, D., Devin, C., Finn, C., Jayaraman, D., & Levine, S. (2019) · arXiv (Cornell University) · Cited by 14
Learn to Effectively Explore in Context-Based Meta-RLPreprint
Zhang, J., Wang, J., Hu, H., Chen, Y., Fan, C.-J., & Zhang, C.-J. (2020) · arXiv.org · doi:10.48550/arxiv.2006.08170 · Cited by 11
“Sun et al. (2011) discusses exploration with information gain, but is restricted to planning problems and requires an oracle estimating the posterior.”
Active Inference, Bayesian Optimal Design, and Expected Utility
Sajid, N., Costa, L. D., Parr, T., & Friston, K. J. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009026949.007 · Cited by 20
Confidence-based progress-driven self-generated goals for skill acquisition in developmental robots
Ngo, H., Luciw, M., Förster, A., & Schmidhuber, J. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00833 · Cited by 13
“Yi et al. (2011) develop a theoretically optimal framework based on the Bayesian meth-ods, in which the agent aims to maximize the information gain in estimating the distribution of model parameters.”
MethodLiakoni, V., Lehmann, M. P., Modirshanechi, A., Brea, J., Lutti, A., Wulfram Gerstner, B., & Preuschoff, K. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118780 · Cited by 16
AutoOD: Automated Outlier Detection via Curiosity-guided Search and Self-imitation LearningPreprint
Li, Y., Chen, Z., Zha, D., Zhou, K., Jin, H., Chen, H., & Hu, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.11321 · Cited by 13
“To address these problems, AutoOD builds on the theory of curiosity-driven exploration [30], aiming to encourage the controller to seek out regions in the searching spaces that are relatively unexplored.”
BackgroundTschantz, A., Baltieri, M., Seth, A. K., & Buckley, C. L. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207382 · Cited by 74
“Reference [67] developed a Bayesian framework in order to maximize information gain via dynamic programming, however, the experiments were limited to discrete state spaces using tabular MDPs.”
MethodBrändle, F., Binz, M., & Schulz, E. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009026949.008 · Cited by 12
Horváth, L., Colcombe, S. J., Milham, M. P., Ray, S., Schwartenbeck, P., & Ostwald, D. (2021) · Computational Brain & Behavior · doi:10.1007/s42113-021-00112-3 · Cited by 12
“Notably, we adopt a Bayesian treatment of exploration and quantify the information bonus as the expected Bayesian surprise (Itti & Baldi, 2009; Sun et al., 2011; Ostwald et al., 2012).”
MethodSun, X., & Bischl, B. (2019) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci44817.2019.9003114 · Cited by 11
“Upon additional observation of action at and state st+1 pair on top of trajectory history ξt, the posterior on the distribution of the environmental parameter θ, p(θ|ξt), could be updated to be p(θ|ξt+1) in a Bayesian way as derived in Equation (43), which is first proposed in [28].”
BackgroundLearning-Driven Exploration for Reinforcement Learning
Usama, M., & Chang, D. E. (2021) · International Conference on Control, Automation and Systems (ICCAS) · doi:10.23919/iccas52745.2021.9649810 · Cited by 9
“strategy based on maximization of information gain about the agent’s belief of the environment dynamics. Using information gain for exploration can be traced to [12] and has been further explored in [13,10,14]. Practical reinforcement learning algorithms often utilize simple exploration heuristics, such as -greedy and…”
Bayesian Brains and the Rényi Divergence
Sajid, N., Faccio, F., Costa, L. D., Parr, T., Schmidhuber, J., & Friston, K. J. (2022) · Neural Computation · doi:10.1162/neco_a_01484 · Cited by 9
Ghazi-Zahedi, K., Martius, G., & Ay, N. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00801 · Cited by 10
New Millennium AI and the Convergence of History: Update of 2012
Schmidhuber, J. (2012) · The frontiers collection · doi:10.1007/978-3-642-32560-1_4 · Cited by 12
“Recent work has demonstrated exploration that is both optimal and feasible [106] for limited scenarios, but much remains to be done for challenging, high-dimensional, partially observable worlds.”
BackgroundContinual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay ProcessesPreprint
Lesort, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00487 · Cited by 31
“When a robot needs to learn a task in a known or unknown environment, it must collect its own training data in the real world Wong (2016). Data serves as the basis for environment exploration and comprehension.”
BackgroundMethodCao, F., & Guo, X. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108663 · Cited by 8
Learning Intrinsic Rewards as a Bi-Level Optimization Problem
Stadie, B. C., Zhang, L., & Ba, J. L. (2020) · Uncertainty in Artificial Intelligence · Cited by 8
Adaptive Procedural Task Generation for Hard-Exploration ProblemsPreprint
Fang, K., Zhu, Y., Savarese, S., & Fei-Fei, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00350 · Cited by 13
“Another family of exploration techniques is derived from an informationtheoretical perspective as maximizing information gain of actions (Houthooft et al., 2016; Sun et al., 2011).”
BackgroundEvaluating Agents without RewardsPreprint
Matusch, B., Ba, J. L., & Hafner, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11538 · Cited by 14
“Information gain has led to successful exploration in RL (Sun et al., 2011; Houthooft et al., 2016; Shyam et al., 2018; Mirchev et al., 2018; Sekar et al., 2020).”
BackgroundMethodVASE: Variational Assorted Surprise Exploration for Reinforcement Learning
Xu, H., Szymanski, L., & McCane, B. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3105140 · Cited by 7
“The main idea is that agents should take actions to maximize the sum of reductions in entropy [19] ∑”
BackgroundExploring Parameter Space with Structured Noise for Meta-Reinforcement Learning
Xu, H., Zhang, C., Wang, J., Ouyang, D., Zheng, Y., & Shao, J. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/436 · Cited by 7
“The concept of curiosity [Sun et al. , 2011] is beneficial to the exploration.”
BackgroundUnburdening the Shoulders of Giants: A Quest for Disconnected Academic Psychology
Krpan, D. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620904775 · Cited by 8
Active Learning for Autonomous Intelligent Agents: Exploration, Curiosity, and InteractionPreprint
Lopes, M. C., & Montesano, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1403.1497 · Cited by 9
“As a generalization of exploration methods in reinforcement learning, such as (Brafman and Tennenholtz, 2003), ideas have been suggested such as plan- ning to be surprised (Sun et al., 2011) or the combination of empirical learning progress with visit counts (Hester and Stone, 2012).”
MethodSchultheis, M., Belousov, B., Abdulsamad, H., & Peters, J. (2019) · Technischen Universität Darmstadt · doi:10.26083/tuprints-00020578 · Cited by 6
An Active Information Seeking Model for Goal-oriented Vision-and-Language Tasks.Preprint
Abbasnejad, E., Wu, Q., Abbasnejad, I., Shi, Q., & van den Hengel, A. (2018) · arXiv (Cornell University) · Cited by 8
Learn to Effectively Explore in Context-Based Meta-RL.Preprint
Zhang, J., Wang, J., Hu, H., Chen, Y., Fan, C., & Zhang, C. (2020) · arXiv (Cornell University) · Cited by 6
Information-based learning by agents in unbounded state spaces
Mobin, S., Arnemann, J., & Sommer, F. (2014) · neural information processing systems · Cited by 5
Computing Machinery, Surprise and Originality
Delacroix, S. (2020) · Philosophy & Technology · doi:10.1007/s13347-021-00453-8 · Cited by 4
“13 Referred to in Taylor (2006). 14 ‘Originality, far from being autonomous, is contingent at every point upon the unoriginal structures that inform it’ (Turner, 1991, p.”
BackgroundImplicit Generative Modeling for Efficient ExplorationPreprint
Ratzlaff, N., Bai, Q., Li, F., & Xu, W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08017 · Cited by 5
“Schmidhuber (1991) and Sun et al. (2011) proposed using the learning progress of the agent as an intrinsic reward.”
Control flow in active inference systemsPreprint
Fields, C., Fabrocini, F., Friston, K. J., Glazebrook, J. F., Hazan, H., Levin, M. G., & Marcianò, A. (2023) · doi:10.31219/osf.io/8e4ra · Cited by 4
Balancing New Against Old Information: The Role of SurprisePreprint
Faraji, M. J., Preuschoff, K., & Wulfram Gerstner, B. (2016) · arXiv (Cornell University) · Cited by 3
A Bandit Framework for Optimal Selection of Reinforcement Learning AgentsPreprint
Merentitis, A., Rasul, K., Vollgraf, R., Sheikh, A.-S., & Bergmann, U. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.03657 · Cited by 3
“so very beneficial, both with respect to focusing early onto the most promising agents, as well as on being able to select the best agent after a relatively small number of steps. It has been shown in [13] that it is beneficial for agents to take actions that maximize the reduction in uncertainty about the environment d…”
BackgroundTowards Informed Exploration for Deep Reinforcement Learning
Tang, H. (2019) · eScholarship (California Digital Library) · Cited by 3
Metadata
- Authors
- Sun, Y., Gomez, F., & Schmidhuber, J.
- Affiliations
- Dalle Molle Institute for Artificial Intelligence Research (CH)
- Venue
- Lecture notes in computer science
- Publisher
- Springer Berlin Heidelberg
- Published
- 2011
- Pages
- 41-51
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W1550989509
- Cited by
- 134
- References
- 11
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 5 · 2023: 11 · 2022: 12 · 2021: 17 · 2020: 30 · 2019: 17 · 2018: 7 · 2017: 6 · 2016: 9 · 2015: 2 · 2014: 3 · 2013: 5 · 2012: 3
- FWCI
- 3.25
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Lecture notes in computer science (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- dynamic environments, artificial general intelligence, surprise, exploration planning, unknown environments
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
