The DOI matches its Crossref record; 10 of the 11 references it cites were found in a registry or an index.
Planning to Be Surprised: Optimal Bayesian Exploration in Dynamic Environments
Open accessOverview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This underwrites Bayesian experimental design [134] and active learning in statistics [135], intrinsic motivation and artificial curiosity in machine learning and robotics [136, 137, 138, 139, 140].”
From The free energy principle made simpler but not too simple (2023) · cites this paper as [139]
In the citing paper
“Adaptive sampling methods are also known in reinforcement learning that can achieve optimal Bayesian exploration of Markov Decision Process environments (Sun et al., 2011; Guez et al., 2012).”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Sun et al., 2011
Citations
Cites
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
Fedorov, V. V., Studden, W. J., & Klimko, E. M. (1972) · Cited by 2,057
Theory of Optimal Experiments.
Herzberg, A. M., Wynn, H. P., Fedorov, V. V., Studden, W. J., & Klimko, E. M. (1972) · Biometrika · doi:10.2307/2334826 · Cited by 1,896 · cited 2×
“In particular, measuring learning progress using KL divergence dates back to the 50’s [2,4].”
BackgroundBayesian Experimental Design: A Review
Chaloner, K. M., & Verdinelli, I. (1995) · Statistical Science · doi:10.1214/ss/1177009939 · Cited by 1,843 · cited 1×
“The idea of actively selecting queries to accelerate learning process has a long history [1, 2, 7], and has received a lot of attention in recent decades, primarily in the context of active learning [8] and artificial curiosity [6].”
BackgroundOn a measure of the information provided by an experiment
Lindley, D. V. (1956) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177728069 · Cited by 1,509 · cited 1×
“In particular, measuring learning progress using KL divergence dates back to the 50’s [2,4].”
BackgroundFormal theory of creativity, fun, and intrinsic motivation (1990–2010)
Schmidhuber, J. (2010) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2010.2056368 · Cited by 747 · cited 2×
“Conceptually, however, this work is closely connected to artificial curiosity and intrinsically motivated reinforcement learning [6,7,9] for agents that actively explore the environment without an external reward signal.”
BackgroundIntrinsically Motivated Reinforcement Learning
Singh, S. P., Barto, A. G., & Chentanez, N. (2005) · doi:10.21236/ada440280 · Cited by 614 · cited 1×
“Conceptually, however, this work is closely connected to artificial curiosity and intrinsically motivated reinforcement learning [6,7,9] for agents that actively explore the environment without an external reward signal.”
BackgroundCurious model-building control systems
Schmidhuber, J. (1991) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1991.170605 · Cited by 601 · cited 2×
“Conceptually, however, this work is closely connected to artificial curiosity and intrinsically motivated reinforcement learning [6,7,9] for agents that actively explore the environment without an external reward signal.”
BackgroundBayesian Surprise Attracts Human Attention
Itti, L., & Baldi, P. (2005) · neural information processing systems · Cited by 363
Planning to Be Surprised: Optimal Bayesian Exploration in Dynamic EnvironmentsPreprint
Sun, Y., Gomez, F. J., & Schmidhuber, J. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.5708 · Cited by 25
Self-citation
Cited by
Sorted by citations
Items 101–131 of 131
Applications of the Free Energy Principle to Machine Learning and NeurosciencePreprint
Millidge, B. (2021) · arXiv (Cornell University) · doi:10.7488/era/1501 · Cited by 13
“Similarly, our approach allows us to understand and rationalize a number of approaches in the literature (Klyubin et al., 2005; Oudeyer & Kaplan, 2009; Sun et al., 2011) which add additional exploratory terms to their reward-maximizing agents, with the heuristic justification that they should increase exploration.”
MethodPathfinding: a neurodynamical account of intuition
Kotler, S., Mannino, M., Friston, K. J., Buzsáki, G., Kelso, J. A. S., & Dumas, G. (2025) · Communications Biology · doi:10.1038/s42003-025-08612-9 · Cited by 6
“The expected informa-tiongainisclearly animportantaspectof intuition, which lendsany planan epistemic affordance 53,133 – 135 .”
Learning One Abstract Bit at a Time Through Self-invented Experiments Encoded as Neural Networks
Herrmann, V., Kirsch, L., & Schmidhuber, J. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-47958-8_16 · Cited by 4
Balancing New Against Old Information: The Role of Surprise in LearningPreprint
Faraji, M., Preuschoff, K., & Gerstner, W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05642 · Cited by 2
“Planning to be surprised so as to maximize information gain has been suggested as an optimal exploration technique in dynamic environments (25, 26).”
MethodHybrid Adversarial Inverse Reinforcement Learning.Preprint
Yuan, M., Pun, M.-O., Chen, Y., & Cao, Q. P. (2021) · arXiv (Cornell University) · Cited by 2
Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for ControlPreprint
Millidge, B., Seth, A. C., & Buckley, C. L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06859 · Cited by 11
“…of fields ranging from machine learning and reinforcement learning (Kaelbling et al., 1998; Oudeyer & Kaplan, 2009; Schmidhuber, 1991, 1999, 2007; Sun et al., 2011; Thrun, 1992; Thrun & Möller, 1991), cognitive science and neuroscience (Cohen et al., 2007; Daw, O’doherty, Dayan, Seymour, &…”
BackgroundParvizi-Wayne, D. (2024) · doi:10.31234/osf.io/b3h2e · Cited by 2
Pragmatic information of aesthetic appraisal
Graben, P. B. (2025) · Cognitive Neurodynamics · doi:10.1007/s11571-025-10225-1 · Cited by 1
The Free Energy Principle and Free Markets
Friston, K. J., Medrano, J., & Verbelen, T. (2026) · Entropy · doi:10.3390/e28090956 · Cited by 1
“This is mathematically the same as expected Bayesian surprise and mutual information that constitutes salience in visual search [97,98].”
Variational Bayesian Exploration-Based Active Sarsa Algorithm
Fu, Q., Yang, Z., Lu, Y., Wu, H., Hu, F., & Chen, J. (2019) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001419510054 · Cited by 1
Distributed and Adaptive Edge-based AI Models for Sensor Networks (DAISeN)
Boeva, V., Casalicchio, E., Abghari, S., Al-Saedi, A. A., Devagiri, V. M., Petef, A., Exner, P., Isberg, A., & Jasarevic, M. (2022) · Annals of Computer Science and Information Systems · doi:10.15439/2022f267 · Cited by 1
“Online change detection algorithms deal with this challenge as it is shown in [24], [25].”
MethodFriston, K. J. (2022) · Physics of Life Reviews · doi:10.1016/j.plrev.2022.07.007 · Cited by 1
Duan, Y. (2017) · eScholarship (California Digital Library) · Cited by 1
Articial Intelligence and Brain Simulation Probes for Interstellar ExpeditionsPreprint
Özkural, E. (2018) · Preprints.org · doi:10.20944/preprints201811.0588.v1 · Cited by 1
“The correct application of curiosity in an AGI controlled interstellar probe seems, therefore, a matter of importance, and thus general theories of curiosity are relevant [36, 42].”
BackgroundThe Intersection of Planning and Learning
Moerland, T. M. (2021) · Research Repository (Delft University of Technology) · doi:10.4233/uuid:5437884e-0078-4b36-b2c7-c6edfea3b418 · Cited by 1
Lesort, T. (2020) · theses.fr (ABES) · doi:10.70675/e714c94ezb6d9z4adbz9daaz60088df88f5f · Cited by 1
On the generation of representations for reinforcement learning
Sun, Y. (2012) · reroDoc Digital Library
Sequential allocation of sampling budgets in unknown environments
Furlong, P. M., & Wettergreen, D. S. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6906603
“This is a sentiment seen in other work, like the optimistic planners of Jurgen Schmidhuber’s group [13], [14], [15], [16].”
ResultInformation Foraging and Change Detection for Automated Science Exploration
Furlong, P. M., & Dille, M. N. (2016) · NASA Technical Reports Server (NASA)
Delacroix, S. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3025626
Position Papers of the 17th Conference on Computer Science and Intelligence Systems
(2022) · Annals of Computer Science and Information Systems · doi:10.15439/978-83-965897-2-9 · Cited by 1
Information-driven intrinsic motivation in reinforcement learning
Ghazi-Zahedi, K., Martius, G., & Ay, N. (2013)
A Derived Functional Framework for Cognitive Computational Learning System
Feng, J., Zhang, & Luo (2026) · Cognitive Computation · doi:10.1007/s12559-026-10575-w
“This is also known as Occam’s principle and is closely related to formulations of universal computation in terms of Solomonov induction and Kolmogorov complexity [39–41].”
Friston, K. J. (2023) · Physics of Life Reviews · doi:10.1016/j.plrev.2023.06.007
Linson, A., Anderson, H., Markham, S., Friston, K. J., & Moutoussis, M. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.07.663473
Learning increasingly complex skills through deep reinforcement learning using intrinsic motivation
Aubret, A. (2021) · theses.fr (ABES) · doi:10.70675/6f863351zdd37z4da4z9fa2z5ceed27ddd07 · Cited by 1
Graph networks as learnable physics engines for inference and control
Sanchez-Gonzalez, A., Heess, N., Springenberg, J. T., Merel, J., Riedmiller, M., Hadsell, R., & Battaglia, P. (2018) · arXiv · doi:10.48550/arxiv.1806.01242 · Cited by 387
“…for learning system identification explicitly (Yu et al., 2017; Peng et al., 2017), learning policies that are robust to hidden property variations (Rajeswaran et al., 2016), and learning exploration strategies in uncertain settings (Schmidhuber, 1991; Sun et al., 2011; Houthooft et al., 2016).”
BackgroundA free energy principle for a particular physics
Friston, K. (2019) · arXiv · doi:10.48550/arxiv.1906.10184 · Cited by 222
“e (Linsker, 1990) and formulations of epistemic foraging in terms of information theory and artificial curiosity (Bialek et al., 2001; Itti and Baldi, 2009; Schmidhuber, 2010; Still and Precup, 2012; Sun et al., 2011; Tishby and Polani, 2010). In the present formulation, internal states come to parameterise beliefs ab…”
BackgroundInfluentialSurprise-Based Intrinsic Motivation for Deep Reinforcement Learning
Achiam, J., & Sastry, S. (2017) · arXiv · doi:10.48550/arxiv.1703.01732 · Cited by 139
“Conceptually and mathematically, our work is closest to prior work on curiosity and surprise [8, 19, 23, 24], although these works focus mainly on small finite MDPs.”
Result
Metadata
- Authors
- Sun, Y., Gomez, F., & Schmidhuber, J.
- Affiliations
- Dalle Molle Institute for Artificial Intelligence Research (CH)
- Venue
- Lecture notes in computer science
- Publisher
- Springer Berlin Heidelberg
- Published
- 2011
- Pages
- 41-51
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W1550989509
- Cited by
- 134
- References
- 11
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 5 · 2023: 11 · 2022: 12 · 2021: 17 · 2020: 30 · 2019: 17 · 2018: 7 · 2017: 6 · 2016: 9 · 2015: 2 · 2014: 3 · 2013: 5 · 2012: 3
- FWCI
- 3.25
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Lecture notes in computer science (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- dynamic environments, artificial general intelligence, surprise, exploration planning, unknown environments
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
