Whence the Expected Free Energy?
Open accessThe DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Abstract
The expected free energy (EFE) is a central quantity in the theory of active inference. It is the quantity that all active inference agents are mandated to minimize through action, and its decomposition into extrinsic and intrinsic value terms is key to the balance of exploration and exploitation that active inference agents evince. Despite its importance, the mathematical origins of this quantity and its relation to the variational free energy (VFE) remain unclear. In this letter, we investigate the origins of the EFE in detail and show that it is not simply ”the free energy in the future.” We present a functional that we argue is the natural extension of the VFE but actively discourages exploratory behavior, thus demonstrating that exploration does not directly follow from free energy minimization into the future. We then develop a novel objective, the free energy of the expected future (FEEF), which possesses both the epistemic component of the EFE and an intuitive mathematical grounding as the divergence between predicted and desired futures.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · cited 10×
“The free-energy principle (FEP) (Friston, 2010; Friston & Ao, 2012; Friston, Kilner, & Harrison, 2006) is an emerging theory from theoretical neuroscience that offers a unifying explanation of the dynamics of self-organizing systems (Friston, 2019; Parr, Da Costa, & Friston, 2020).”
BackgroundInfluentialA theory of cortical responses.
Friston, K. (2005) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2005.1622 · Cited by 4,578
The Bayesian brain: The role of uncertainty in neural coding and computation
Knill, D. C., & Pouget, A. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.10.007 · Cited by 2,588 · cited 1×
“…(K. Friston, 2003, 2005, 2008; K. J. Friston, 2008; K. J. Friston, Trujillo-Barreto, & Daunizeau, 2008), extending the Bayesian Brain hypothesis (Deneve, 2005; Doya, Ishii, Pouget, & Rao, 2007; Knill & Pouget, 2004) by postulating that the neural dynamics of the brain perform variational inference.”
Canonical Microcircuits for Predictive Coding
Bastos, A. M., Usrey, W. M., Adams, R. A., Mangun, G. R., Fries, P., & Friston, K. J. (2012) · Neuron · doi:10.1016/j.neuron.2012.10.038 · Cited by 2,421 · cited 2×
“Under certain assumptions about the forms of the densities embodied by the agent, this theory can even be translated down to the level of neural circuits in the form of a biologically plausible neuronal process theory (Bastos et al., 2012; Friston, 2008a; Kanai, Komura, Shipp, & Friston, 2015; Shipp, 2016; Spratling,…”
BackgroundCuriosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815
A free energy principle for the brain
Friston, K., Kilner, J., & Harrison, L. (2006) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2006.10.001 · Cited by 1,369
Bayesian surprise attracts human attention
Itti, L., & Baldi, P. (2009) · Vision Research · doi:10.1016/j.visres.2008.09.007 · Cited by 1,224
Active inference: a process theory
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., & Pezzulo, G. (2017) · Neural Computation · doi:10.1162/neco_a_00912 · Cited by 1,156
Hierarchical Models in the Brain
Friston, K. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000211 · Cited by 952 · cited 7×
“The generalized coordinates mean that the agent is effectively performing variational inference over a Taylor-expanded future trajectory instead of a temporally instant hidden state (Friston, 2008a; Friston et al., 2008).”
BackgroundInfluentialWhat is intrinsic motivation? A typology of computational approaches
Oudeyer, P.-Y. (2007) · Frontiers in Neurorobotics · doi:10.3389/neuro.12.006.2007 · Cited by 811
Active inference and epistemic value
Friston, K., Rigoli, F., Ognibene, D., Mathys, C., Fitzgerald, T., & Pezzulo, G. (2015) · Cognitive Neuroscience · doi:10.1080/17588928.2015.1020053 · Cited by 731
Learning and inference in the brain
Friston, K. (2003) · Neural Networks · doi:10.1016/j.neunet.2003.06.005 · Cited by 724 · cited 1×
“tionalinferencewhich minimizesa single information-theoreticobjective– the Variational Free-Energy (VFE). In theoretical neuroscience, the FEP translates into an elegant account of brain function (K. Friston, 2003, 2005, 2008; K. J. Friston, 2008; K. J. Friston, Trujillo-Barreto,& Daunizeau, 2008), extending theBa…”
BackgroundFriston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., ODoherty, J., & Pezzulo, G. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.06.022 · Cited by 674
Action understanding and active inference
Friston, K., Mattout, J., & Kilner, J. (2011) · Biological Cybernetics · doi:10.1007/s00422-011-0424-z · Cited by 665
Reinforcement learning or active inference?
Friston, K. J., Daunizeau, J., & Kiebel, S. J. (2009) · PLoS ONE · doi:10.1371/journal.pone.0006421 · Cited by 443
Cerebral hierarchies: predictive processing, precision and the pulvinar
Kanai, R., Komura, Y., Shipp, S., & Friston, K. (2015) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2014.0169 · Cited by 432
The graphical brain: Belief propagation and active inference
Friston, K. J., Parr, T., & de Vries, B. (2017) · Network Neuroscience · doi:10.1162/netn_a_00018 · Cited by 415
Large-Scale Study of Curiosity-Driven Learning
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., & Efros, A. A. (2018) · arXiv · doi:10.48550/arxiv.1808.04355 · Cited by 401 · cited 1×
“The fact that intrinsic drives naturally emerge from this formulation is argued as an advantage over other formulations that typically encourage exploration by adding ad hoc exploratory terms to their loss function (Burda et al., 2018; Mohamed & Rezende, 2015; Oudeyer & Kaplan, 2009; Pathak, Agrawal, Efros, & Darrell,…”
BackgroundReinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1805.00909 · Cited by 366 · cited 2×
“The intuition behind the FEEF as a divergence between desired and expected future observations is also similar to probabilistic formulations of the reinforcement learning problem (Attias, 2003; Kappen, 2005; Levine, 2018; Toussaint, 2009), which typically try to minimize the divergence between a controlled trajectory,…”
BackgroundNeural elements for predictive coding
Shipp, S. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01792 · Cited by 340 · cited 1×
“to the levelof neural circuits in the formof a A PREPRINT - SEPTEMBER 30, 2020 biologicallyplausibleneuronalprocesstheory(Bastos et al.,2012;K. Friston,2008;Kanai, Komura, Shipp, & Friston, 2015; Shipp, 2016; Spratling, 2008). Action is then subsumed into this formulation,under the name of active inference (K. Fri…”
BackgroundWhat is optimal about motor control?
Friston, K. (2011) · Neuron · doi:10.1016/j.neuron.2011.10.018 · Cited by 332
Path integrals and symmetry breaking for optimal control theory
Kappen, H. J. (2005) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/2005/11/p11011 · Cited by 309 · cited 2×
“…between desired and expected future observations is also similar to probabilistic formulations of the reinforcement learning problem (Attias, 2003; Kappen, 2005; Levine, 2018; Toussaint, 2009), which typically try to minimize the divergence between a controlled trajectory, and an optimal…”
BackgroundDEM: A variational treatment of dynamic systems
Friston, K. J., Trujillo-Barreto, N., & Daunizeau, J. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.02.054 · Cited by 278 · cited 1×
“The generalized coordinates mean that the agent is effectively performing variational inference over a Taylor-expanded future trajectory instead of a temporally instant hidden state (Friston, 2008a; Friston et al., 2008).”
BackgroundUncertainty, epistemics and active inference
Parr, T., & Friston, K. J. (2017) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0376 · Cited by 252
Generalised free energy and active inference
Parr, T., & Friston, K. J. (2019) · Biological Cybernetics · doi:10.1007/s00422-019-00805-w · Cited by 236
A free energy principle for a particular physics
Friston, K. (2019) · arXiv · doi:10.48550/arxiv.1906.10184 · Cited by 222 · cited 4×
“The free-energy principle (FEP) (Friston, 2010; Friston & Ao, 2012; Friston, Kilner, & Harrison, 2006) is an emerging theory from theoretical neuroscience that offers a unifying explanation of the dynamics of self-organizing systems (Friston, 2019; Parr, Da Costa, & Friston, 2020).”
BackgroundInfluentialAn information-theoretic approach to curiosity-driven reinforcement learning
Still, S., & Precup, D. (2012) · Theory in Biosciences · doi:10.1007/s12064-011-0142-z · Cited by 165
Scene Construction, Visual Foraging, and Active Inference
Mirza, M. B., Adams, R. A., Mathys, C. D., & Friston, K. J. (2016) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2016.00056 · Cited by 165
Reconciling predictive coding and biased competition models of cortical function
Spratling, M. W. (2008) · Frontiers in Computational Neuroscience · doi:10.3389/neuro.10.004.2008 · Cited by 153
Markov blankets, information geometry and stochastic thermodynamics
Parr, T., Da Costa, L., & Friston, K. (2020) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2019.0159 · Cited by 141 · cited 2×
“Finally, it is important to note that although in this paper we have solely been concerned with the EFE and active inference in discrete-time POMDPs, the original intuitions and mathematical framework of the free-energy principle arose out of a continuous time formulation, deeply interwoven with concerns from informat…”
BackgroundPlanning to be surprised: optimal bayesian exploration in dynamic environments
Sun, Y., Gomez, F., & Schmidhuber, J. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22887-2_5 · Cited by 134
Neuronal message passing using mean-field, bethe, and marginal approximations
Parr, T., Markovic, D., Kiebel, S. J., & Friston, K. J. (2019) · Scientific Reports · doi:10.1038/s41598-018-38246-3 · Cited by 128
Deep temporal models and active inference
Friston, K. J., Rosch, R., Parr, T., Price, C., & Bowman, H. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.04.004 · Cited by 119
Ueltzhöffer, K. (2018) · Biological Cybernetics · doi:10.1007/s00422-018-0785-7 · Cited by 98 · cited 1×
“rr & Friston, 2017a, 2018a), solving reinforcement learning benchmarks (Çatal, Verbelen,Nauta, De Boom, & Dhoedt, 2020; Millidge, 2019a, 2019b; Tschantz, Baltieri, Seth, Buckley,et al., 2019; Ueltzhöffer, 2018; van de Laar & de Vries, 2019), to modelling psychiatric disorders as cases of aberrant inference…”
Cited by
A step-by-step tutorial on active inference and its application to empirical data
Smith, R., Friston, K. J., & Whyte, C. J. (2022) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2021.102632 · Cited by 246
Predictive processing as a systematic basis for identifying the neural correlates of consciousness
Hohwy, J., & Seth, A. (2020) · Philosophy and the Mind Sciences · doi:10.33735/phimisci.2020.ii.64 · Cited by 170
(2021) · UCL Discovery (University College London) · Cited by 141
Interoception as modeling, allostasis as control
Sennesh, E., Theriault, J., Brooks, D., van de Meent, J.-W., Barrett, L. F., & Quigley, K. S. (2022) · Biological Psychology · doi:10.1016/j.biopsycho.2021.108242 · Cited by 107
The empirical status of predictive coding and active inference
Hodson, R., Mehta, M., & Smith, R. (2024) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105473 · Cited by 69
Metadata
- Authors
- Millidge, B., Tschantz, A., & Buckley, C. L.
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2021-02-01
- Volume
- 33
- Issue
- 2
- Pages
- 447-482
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W3017237277
- Cited by
- 94
- References
- 34
- FWCI
- 6.59
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.