What is value—accumulated reward or evidence?
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Why are you reading this abstract? In some sense, your answer will cast the exercise as valuable-but what is value? In what follows, we suggest that value is evidence or, more exactly, log Bayesian evidence. This implies that a sufficient explanation for valuable behavior is the accumulation of evidence for internal models of our world. This contrasts with normative models of optimal control and reinforcement learning, which assume the existence of a value function that explains behavior, where (somewhat tautologically) behavior maximizes value. In this paper, we consider an alternative formulation-active inference-that replaces policies in normative models with prior beliefs about the (future) states agents should occupy. This enables optimal behavior to be cast purely in terms of inference: where agents sample their sensorium to maximize the evidence for their generative model of hidden states in the world, and minimize their uncertainty about those states. Crucially, this formulation resolves the tautology inherent in normative models and allows one to consider how prior beliefs are themselves optimized in a hierarchical setting. We illustrate these points by showing that any optimal policy can be specified with prior beliefs in the context of Bayesian inference. We then show how these prior beliefs are themselves prescribed by an imperative to minimize uncertainty. This formulation explains the saccadic eye movements required to read this text and defines the value of the visual sensations you are soliciting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · cited 5×
“Surprise is minimized—approximately or exactly—if agents minimize a variational free energy bound on surprise (Feynman, 1972; Hinton and van Camp, 1993), given a generative model m of state transitions (Dayan et al., 1995; Friston, 2010).”
BackgroundInfluentialSelf-citationEmergence of simple-cell receptive field properties by learning a sparse code for natural images
Olshausen, B. A., & Field, D. J. (1996) · Nature · doi:10.1038/381607a0 · Cited by 5,694
Rao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483 · cited 3×
“The first is known as generalized predictive coding and has the same form as Bayesian (e.g., Kalman–Bucy) filters used in time series analysis; see also Rao and Ballard (1999).”
BackgroundMethodComputational modelling of visual attention
Itti, L., & Koch, C. (2001) · Nature Reviews Neuroscience · doi:10.1038/35058500 · Cited by 4,733
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149
Gigerenzer, G., & Gaissmaier, W. (2011) · Annual Review of Psychology · doi:10.1146/annurev-psych-120709-145346 · Cited by 4,037
Yarbus, A. L. (1967) · doi:10.1007/978-1-4899-5379-7 · Cited by 3,304
A sensorimotor account of vision and visual consciousness
O'Regan, J. K., & Noë, A. (2001) · Behavioral and Brain Sciences · doi:10.1017/s0140525x01000115 · Cited by 3,126
A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants
Neal, R. M., & Hinton, G. E. (1998) · Learning in Graphical Models · doi:10.1007/978-94-011-5014-9_12 · Cited by 2,187
The free-energy principle: a rough guide to the brain?
Friston, K. (2009) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2009.04.005 · Cited by 1,831
Self-citationPredictive coding under the free-energy principle
Friston, K., & Kiebel, S. (2009) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2008.0300 · Cited by 1,579
Self-citationToward a modern theory of adaptive networks: Expectation and prediction.
Sutton, R. S., & Barto, A. G. (1981) · Psychological Review · doi:10.1037/0033-295x.88.2.135 · Cited by 1,450 · cited 2×
“In these schemes the value function is itself learnt (Rescorla and Wagner, 1972; Sutton and Barto, 1981; Watkins and Dayan, 1992; Friston et al., 1994): This enables one to solve Markov decision problems without learning the transition probabilities, because the value function acts as a guidance function for action.”
BackgroundA free energy principle for the brain
Friston, K., Kilner, J., & Harrison, L. (2006) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2006.10.001 · Cited by 1,369
Self-citationAttention, uncertainty, and free-energy
Feldman, H., & Friston, K. J. (2010) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2010.00215 · Cited by 1,365
Self-citationBayesian surprise attracts human attention
Itti, L., & Baldi, P. (2009) · Vision Research · doi:10.1016/j.visres.2008.09.007 · Cited by 1,224
On the computational architecture of the neocortex: II The role of cortico-cortical loops
Mumford, D. (1992) · Biological Cybernetics · doi:10.1007/bf00198477 · Cited by 1,074
Attention, intention, and priority in the parietal lobe
Bisley, J. W., & Goldberg, M. E. (2010) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-060909-152823 · Cited by 1,043 · cited 2×
“Crucially, the predictions of visual input depend upon posterior beliefs about the direction of gaze, encoded by the state units in the “where” stream (Bisley and Goldberg, 2010).”
BackgroundOn the theory of dynamic programming
Bellman, R. (1952) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.38.8.716 · Cited by 998
Keeping the neural networks simple by minimizing the description length of the weights
Hinton, G. E., & van Camp, D. (1993) · Proceedings of the sixth annual conference on Computational learning theory - COLT '93 · doi:10.1145/168304.168306 · Cited by 994 · cited 1×
“We will see later that formulating the optimal control problem in terms of prior beliefs enables us to connect to other normative theories about perception and entertain questions about where these prior beliefs come from.”
BackgroundHierarchical Models in the Brain
Friston, K. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000211 · Cited by 952 · cited 2×
“From now on, the tilde notation x̃ = (x, x′, x′′, . . .) denotes variables in generalized coordinates of motion (Friston, 2008), where each prime denotes a temporal derivative (using Lagrange’s notation).”
BackgroundMethodSelf-citationOn the computational architecture of the neocortex: I. The role of the thalamo-cortical loop
Mumford, D. (1991) · Biological Cybernetics · doi:10.1007/bf00202389 · Cited by 929
Birkhoff, G. D. (1931) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.17.2.656 · Cited by 867
Action and behavior: a free-energy formulation
Friston, K. J., Daunizeau, J., Kilner, J., & Kiebel, S. J. (2010) · Biological Cybernetics · doi:10.1007/s00422-010-0364-z · Cited by 866 · cited 7×
“In brief, these simulations use differential equations that minimize the free energy of sensory input using a generalized gradient descent (Friston et al., 2010a).”
BackgroundMethodInfluentialSelf-citationVision as Bayesian inference: analysis by synthesis?
Yuille, A., & Kersten, D. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.05.002 · Cited by 849
Gregory, R. L. (1980) · Philosophical Transactions of the Royal Society of London. B, Biological Sciences · doi:10.1098/rstb.1980.0090 · Cited by 789
Eye guidance in natural vision: reinterpreting salience
Tatler, B. W., Hayhoe, M. M., Land, M. F., & Ballard, D. H. (2011) · Journal of Vision · doi:10.1167/11.5.5 · Cited by 775 · cited 1×
“Saccadic eye movements are a useful vehicle to illustrate active inference because they speak directly to visual search strategies and a wealth of psychophysical, neurobiological, and theoretical study (e.g., Grossberg et al., 1997; Ferreira et al., 2008; Srihasam et al., 2009; Bisley and Goldberg, 2010; Shires et al.…”
BackgroundAction understanding and active inference
Friston, K., Mattout, J., & Kilner, J. (2011) · Biological Cybernetics · doi:10.1007/s00422-011-0424-z · Cited by 665
Self-citationDecision theory, reinforcement learning, and the brain
Dayan, P., & Daw, N. D. (2008) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.8.4.429 · Cited by 580
Perceptions as Hypotheses: Saccades as Experiments
Friston, K., Adams, R. A., Perrinet, L., & Breakspear, M. (2012) · Frontiers in Psychology · doi:10.3389/fpsyg.2012.00151 · Cited by 462 · cited 1×
“The anticipated trajectories are the expected values based upon posterior expectations about past and future states.”
BackgroundSelf-citationReinforcement learning or active inference?
Friston, K. J., Daunizeau, J., & Kiebel, S. J. (2009) · PLoS ONE · doi:10.1371/journal.pone.0006421 · Cited by 443
Self-citationVariance as a signature of neural computations during decision making
Churchland, A. K., Kiani, R., Chaudhuri, R., Wang, X.-J., Pouget, A., & Shadlen, M. N. (2011) · Neuron · doi:10.1016/j.neuron.2010.12.037 · Cited by 350
What is optimal about motor control?
Friston, K. (2011) · Neuron · doi:10.1016/j.neuron.2011.10.018 · Cited by 332
Self-citationPath integrals and symmetry breaking for optimal control theory
Kappen, H. J. (2005) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/2005/11/p11011 · Cited by 309
Tani, J., & Nolfi, S. (1999) · Neural Networks · doi:10.1016/s0893-6080(99)00060-x · Cited by 259
General duality between optimal control and estimation
Todorov, E. (2008) · 2008 47th IEEE Conference on Decision and Control · doi:10.1109/cdc.2008.4739438 · Cited by 248
Active inference and agency: optimal control without cost functions
Friston, K., Samothrakis, S., & Montague, R. (2012) · Biological Cybernetics · doi:10.1007/s00422-012-0512-8 · Cited by 241
Self-citationFriston, K., Stephan, K., Li, B., & Daunizeau, J. (2010) · Mathematical Problems in Engineering · doi:10.1155/2010/621670 · Cited by 171
Self-citationThalamic pathways for active vision
Wurtz, R. H., McAlonan, K., Cavanaugh, J., & Berman, R. A. (2011) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2011.02.004 · Cited by 127 · cited 4×
“This sampling depends upon action or an embodied inference that speaks directly to the notion of active vision or visual palpation (O’Regan and Noë, 2001; Wurtz et al., 2011).”
BackgroundInfluentialA complete class theorem for statistical problems with finite sample spaces
Brown, L. D. (1981) · The Annals of Statistics · doi:10.1214/aos/1176345645 · Cited by 97
Cited by
Active inference: a process theory
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., & Pezzulo, G. (2017) · Neural Computation · doi:10.1162/neco_a_00912 · Cited by 1,156
Self-citationFriston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., ODoherty, J., & Pezzulo, G. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.06.022 · Cited by 674
Self-citationAllostatic Self-efficacy: A Metacognitive Theory of Dyshomeostasis-Induced Fatigue and Depression
Stephan, K. E., Manjaly, Z. M., Mathys, C. D., Weber, L. A. E., Paliwal, S., Gard, T., Tittgemeyer, M., Fleming, S. M., Haker, H., Seth, A. K., et al. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00550 · Cited by 437
“This, however, only shifts the problem and raises the question what constitutes “reward” for the brain (compare the discussion in Friston et al., 2012).”
BackgroundActive Inference, Curiosity and Insight
Friston, K. J., Lin, M., Frith, C. D., Pezzulo, G., Hobson, J. A., & Ondobaka, S. (2017) · Neural Computation · doi:10.1162/neco_a_00999 · Cited by 351
Self-citationGeneralised free energy and active inference
Parr, T., & Friston, K. J. (2019) · Biological Cybernetics · doi:10.1007/s00422-019-00805-w · Cited by 236
“Recently, we have migrated the basic idea from models that generate continuous sensations (like velocity and luminance contrast) (Brown and Friston 2012) to discrete state-space models; specifically Markov decision processes (Friston et al. 2017a).”
BackgroundSelf-citation
Metadata
- Authors
- Friston, K., Adams, R., & Montague, R.
- Venue
- Frontiers in Neurorobotics
- Publisher
- Frontiers Media SA
- Published
- 2012
- Volume
- 6
- Pages
- 11-11
- ISSN
- 1662-5218 (ISSN-L)
- OpenAlex
- W2145572963
- Cited by
- 67
- References
- 39
- FWCI
- 0.77
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.