The DOI matches its Crossref record; 98 of the 98 references it cites were found in a registry or an index.
Computational mechanisms of curiosity and goal-directed exploration
Open accessAbstract
Successful behaviour depends on the right balance between maximising reward and soliciting information about the world. Here, we show how different types of information-gain emerge when casting behaviour as surprise minimisation. We present two distinct mechanisms for goal-directed exploration that express separable profiles of active sampling to reduce uncertainty. ‘Hidden state’ exploration motivates agents to sample unambiguous observations to accurately infer the (hidden) state of the world. Conversely, ‘model parameter’ exploration, compels agents to sample outcomes associated with high uncertainty, if they are informative for their representation of the task structure. We illustrate the emergence of these types of information-gain, termed active inference and active learning, and show how these forms of exploration induce distinct patterns of ‘Bayes-optimal’ behaviour. Our findings provide a computational framework for understanding how distinct levels of uncertainty systematically affect the exploration-exploitation trade-off in decision-making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Reinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,414
THOMPSON, W. R. (1933) · Biometrika · doi:10.1093/biomet/25.3-4.285 · Cited by 2,712
Cortical substrates for exploratory decisions in humans
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006) · Nature · doi:10.1038/nature04766 · Cited by 2,224
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 1×
“In addition, dopamine neurons have been shown to encode ‘infor-mation prediction errors’ analogously to ‘reward prediction errors’ ( Montague et al., 1996 ; Schultz et al., 1997 ).”
BackgroundInformation-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
Srinivas, N., Krause, A., Kakade, S. M., & Seeger, M. W. (2012) · IEEE Transactions on Information Theory · doi:10.1109/tit.2011.2182033 · Cited by 1,850
Uncertainty, Neuromodulation, and Attention
Yu, A. J., & Dayan, P. (2005) · Neuron · doi:10.1016/j.neuron.2005.04.026 · Cited by 1,774 · cited 2×
“These types of uncertainties have been termed unexpected and expected uncertainty ( Yu and Dayan, 2005 ) or, in economics, ambiguity and risk.”
Background(2003) · Cited by 1,680
Cited by
In vitro neurons learn and exhibit sentience when embodied in a simulated game-world
Kagan, B. J., Kitchen, A. C., Tran, N. T., Habibollahi, F., Khajehnejad, M., Parker, B. J., Bhat, A., Rollo, B., Razi, A., & Friston, K. J. (2022) · Neuron · doi:10.1016/j.neuron.2022.09.001 · Cited by 400
Self-citationA step-by-step tutorial on active inference and its application to empirical data
Smith, R., Friston, K. J., & Whyte, C. J. (2022) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2021.102632 · Cited by 246
Self-citationCorcoran, A. W., Pezzulo, G., & Hohwy, J. (2020) · Biology & Philosophy · doi:10.1007/s10539-020-09746-2 · Cited by 161
(2021) · UCL Discovery (University College London) · Cited by 141
Learning action-oriented models through active inference
Tschantz, A., Seth, A. K., & Buckley, C. L. (2020) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007805 · Cited by 121
Deep active inference as variational policy gradients
Millidge, B. (2020) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2020.102348 · Cited by 115
Active inference as a theory of sentient behavior
Pezzulo, G., Parr, T., & Friston, K. (2024) · Biological Psychology · doi:10.1016/j.biopsycho.2023.108741 · Cited by 109
Self-citationCuriosity as a Metacognitive Feeling
Goupil, L., & Proust, J. (2023) · Cognition · doi:10.1016/j.cognition.2022.105325 · Cited by 89
Dogmatism manifests in lowered information search under uncertainty
Schulz, L., Rollwage, M., Dolan, R. J., & Fleming, S. M. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2009641117 · Cited by 76
“In contrast, normative frameworks propose that uncertainty, rather than valence, should determine where and when we should seek information (5, 7, 15).”
BackgroundCarruthers, P., & Williams, D. M. (2022) · Cognition · doi:10.1016/j.cognition.2022.105117 · Cited by 71
“And although some work modeling curiosity-like behavior has been done in this framework (Friston et al., 2017; Schwartenbeck et al., 2019), no one has, to the best of our knowledge, discussed curiosity following recognition failure, nor done the sort of detailed testing-and-modeling using…”
BackgroundDeep active inference agents using Monte-Carlo methods
Fountas, Z., Sajid, N., Mediano, P. A. M., & Friston, K. (2020) · arXiv · doi:10.48550/arxiv.2006.04176 · Cited by 64
Self-citationEpistemic Communities under Active Inference
Albarracin, M., Demekas, D., Ramstead, M. J. D., & Heins, C. (2022) · Entropy · doi:10.3390/e24040476 · Cited by 60
“in [73,126]) and update the 882 posterior habit vector ε at every timestep, i.”
BackgroundPredictive processing in cognitive robotics: A review
Ciria, A., Schillaci, G., Pezzulo, G., Hafner, V. V., & Lara, B. (2021) · Neural Computation · doi:10.1162/neco_a_01383 · Cited by 51
“Studies such as those from Tschantz et al. (2020), Friston et al. (2015), Schwartenbeck et al. (2019) and Schillaci et al. (2020a) have discussed epistemic and emotional value, homeostatic drives and intrinsic motivation that regulate behaviors.”
BackgroundActive Inference and Behavior Trees for Reactive Action Planning and Execution in Robotics
Pezzato, C., Corbato, C. H., Bonhof, S., & Wisse, M. (2023) · IEEE Transactions on Robotics · doi:10.1109/tro.2022.3226144 · Cited by 44
Affective experience in the predictive mind: a review and new integrative account
Fernandez Velasco, P., & Loev, S. (2021) · Synthese · doi:10.1007/s11229-020-02755-4 · Cited by 43
Neural dynamics under active inference: plausibility and efficiency of information processing
Da Costa, L., Parr, T., Sengupta, B., & Friston, K. (2021) · Entropy · doi:10.3390/e23040454 · Cited by 38
Self-citationReward maximization through discrete active inference
Da Costa, L., Sajid, N., Parr, T., Friston, K., & Smith, R. (2023) · Neural Computation · doi:10.1162/neco_a_01574 · Cited by 35
Self-citationFederated inference and belief sharing
Friston, K. J., Parr, T., Heins, C., Constant, A., Friedman, D., Isomura, T., Fields, C., Verbelen, T., Ramstead, M., Clippinger, J., et al. (2024) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105500 · Cited by 30
Self-citationIs free-energy minimisation the mark of the cognitive?
Kiverstein, J., & Sims, M. (2021) · Biology & Philosophy · doi:10.1007/s10539-021-09788-0 · Cited by 23
“dyshomeostasis expected as a consequence of action (cf. Friston et al. 2015; Schwartenbeck et al. 2019).”
BackgroundCultivating creativity: predictive brains and the enlightened room problem
Constant, A., Friston, K. J., & Clark, A. (2024) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2022.0415 · Cited by 19
Self-citationOn the minimal theory of consciousness implicit in active inference
Whyte, C. J., Corcoran, A. W., Robinson, J., Smith, R., Moran, R. J., Parr, T., Friston, K. J., Seth, A. K., & Hohwy, J. (2026) · Physics of Life Reviews · doi:10.1016/j.plrev.2025.11.002 · Cited by 15
Self-citationExploration and preference satisfaction trade-off in reward-free learning
Sajid, N., Tigas, P., Zakharov, A., Fountas, Z., & Friston, K. (2021) · arXiv · doi:10.48550/arxiv.2106.04316 · Cited by 11
“The first one instantiates preference learning over reward outcomes, as presented below for a single time instance τ [54, 14]:”
BackgroundSelf-citationEmbodied object representation learning and recognition
Van de Maele, T., Verbelen, T., Çatal, O., & Dhoedt, B. (2022) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2022.840658 · Cited by 9
“Information gain on model parameters is, in the active inference literature, called “novelty”, while information gain on latent states or attributes is associated with “salience” (Schwartenbeck et al., 2019).”
BackgroundDeconstructing Deep Active Inference: A Contrarian Information Gatherer
Champion, T., Grześ, M., Bonheme, L., & Bowman, H. (2024) · Neural Computation · doi:10.1162/neco_a_01697 · Cited by 5
Revealing human sensitivity to a latent temporal structure of changes
Marković, D., Reiter, A. M. F., & Kiebel, S. J. (2022) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2022.962494 · Cited by 2
Metadata
- Authors
- Schwartenbeck, P., Passecker, J., Hauser, T. U., FitzGerald, T. H., Kronbichler, M., & Friston, K. J.
- Venue
- eLife
- Publisher
- eLife Sciences Publications, Ltd
- Published
- 2019-05-10
- Volume
- 8
- ISSN
- 2050-084X (ISSN-L)
- OpenAlex
- W2892266804
- Cited by
- 254
- References
- 98
- FWCI
- 14.04
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.