Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 84 of the 84 references it cites were found in a registry or an index.

Active inference and agency: optimal control without cost functions

Open access
Friston, Karl, Samothrakis, Spyridon, Montague, Read · 2012 · Biological Cybernetics

Abstract

This paper describes a variational free-energy formulation of (partially observable) Markov decision problems in decision making under uncertainty. We show that optimal control can be cast as active inference. In active inference, both action and posterior beliefs about hidden states minimise a free energy bound on the negative log-likelihood of observed states, under a generative model. In this setting, reward or cost functions are absorbed into prior beliefs about state transitions and terminal states. Effectively, this converts optimal control into a pure inference problem, enabling the application of standard Bayesian filtering techniques. We then consider optimal trajectories that rest on posterior beliefs about hidden states in the future. Crucially, this entails modelling control as a hidden state that endows the generative model with a representation of agency. This leads to a distinction between models with and without inference on hidden control states; namely, agency-free and agency-based models, respectively.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s00422-012-0512-8

Cites

84 of 84 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Probabilistic graphical model illustrating the Markovian dependencies among hidden states generating sensory data. These hidden states ( st , ut ) are represented explicitly, over all time points: t = 1 , . . . , T . This means there is a representation of the past and future that includes hidden states mediating control. Note that the dependency of this hidden Markov model runs backwards in time so that all preceding hidden states are conditioned recursively on the final or terminal goal state
Fig. 1 Probabilistic graphical model illustrating the Markovian dependencies among hidden states generating sensory data. These hidden states ( st , ut ) are represented explicitly, over all time points: t = 1 , . . . , T . This means there is a representation of the past and future that includes hidden states mediating control. Note that the dependency of this hidden Markov model runs backwards in time so that all preceding hidden states are conditioned recursively on the final or terminal goal stateCC-BY · From the PDF
Fig. 2 Schematic of the mountain car problem: the upper panel (and associated equations) illustrate the landscape or potential energy function that defines the motion of the car. This has a minima at x = -0 . 5. The mountain-car is shown at the desired parking position at the top of the hill on the right x = 1 (indicated with a red ball ). The equations of motion in the lower panel describe the forces exerted on the car, which include σ( a ) , a sigmoid (hyperbolic tangent) function of action, gravitational forces and friction.
Fig. 2 Schematic of the mountain car problem: the upper panel (and associated equations) illustrate the landscape or potential energy function that defines the motion of the car. This has a minima at x = -0 . 5. The mountain-car is shown at the desired parking position at the top of the hill on the right x = 1 (indicated with a red ball ). The equations of motion in the lower panel describe the forces exerted on the car, which include σ( a ) , a sigmoid (hyperbolic tangent) function of action, gravitational forces and friction.CC-BY · From the PDF
Fig. 3 Illustration of the transitional probabilities describing the discrete formulation of the mountain car problem in the previous figure. These correspond to the probability of occupying a hidden state given a particular final state (fi rst column ), after four time steps ( second column ) and after eight time steps into the past. The probability distribution functions are shown in image format, over position and velocity. The top row shows the transition probabilities for a control that applies leftward forces to the mountain car, the middle row shows the corresponding pullback probabilities for a control that exerts no forces and the last row shows transition probabilities for control states that accelerate the car towards the right. Note the dispersion of these probability distributions over time, due to the discretisation of the dynamics and random fluctuations on the speed and position
Fig. 3 Illustration of the transitional probabilities describing the discrete formulation of the mountain car problem in the previous figure. These correspond to the probability of occupying a hidden state given a particular final state (fi rst column ), after four time steps ( second column ) and after eight time steps into the past. The probability distribution functions are shown in image format, over position and velocity. The top row shows the transition probabilities for a control that applies leftward forces to the mountain car, the middle row shows the corresponding pullback probabilities for a control that exerts no forces and the last row shows transition probabilities for control states that accelerate the car towards the right. Note the dispersion of these probability distributions over time, due to the discretisation of the dynamics and random fluctuations on the speed and positionCC-BY · From the PDF
Fig. 4 This figure shows the results of a simulated (agency based) trajectory over T = 16 time steps starting at x = ( 0 , 0 ) and ending at the goal location x = ( 1 , 0 ) ( red ball ) using active inference and explicit representations of the future. The upper panel shows the trajectories in the state space of position and velocity. The grey lines represent anticipated trajectories accumulated during control, while the dark ( dotted ) lines show the actual trajectory through state space. The anticipated trajectories are the expected values based upon posterior expectations about past and future states. They are therefore continuous functions of position and velocity. In contrast, the actual trajectory is restricted to the 1,024 discrete states that can be occupied; these are shown as light grey dots . The lower panels show the anticipated control and the actual actions selected under active inference (in image format where lighter colours mean a higher probability). Note that there is a high degree of correspondence; however, the posterior beliefs about control are not always absolutely certain: these are the beliefs at the times each action is selected
Fig. 4 This figure shows the results of a simulated (agency based) trajectory over T = 16 time steps starting at x = ( 0 , 0 ) and ending at the goal location x = ( 1 , 0 ) ( red ball ) using active inference and explicit representations of the future. The upper panel shows the trajectories in the state space of position and velocity. The grey lines represent anticipated trajectories accumulated during control, while the dark ( dotted ) lines show the actual trajectory through state space. The anticipated trajectories are the expected values based upon posterior expectations about past and future states. They are therefore continuous functions of position and velocity. In contrast, the actual trajectory is restricted to the 1,024 discrete states that can be occupied; these are shown as light grey dots . The lower panels show the anticipated control and the actual actions selected under active inference (in image format where lighter colours mean a higher probability). Note that there is a high degree of correspondence; however, the posterior beliefs about control are not always absolutely certain: these are the beliefs at the times each action is selectedCC-BY · From the PDF
Fig. 5 This figure uses the same format as the previous figure. The only difference here is that we have changed the location of the goal to x = ( 1 , -1 ) . In other words, we require the mountain car to pass through the goal location from the right, with unit speed. It can be seen that this goal has been obtained, with reasonable accuracy
Fig. 5 This figure uses the same format as the previous figure. The only difference here is that we have changed the location of the goal to x = ( 1 , -1 ) . In other words, we require the mountain car to pass through the goal location from the right, with unit speed. It can be seen that this goal has been obtained, with reasonable accuracyCC-BY · From the PDF
Fig. 6 This figure reports the same results as shown in Fig. 4; however, here we have increased the number of time steps from T = 16 to T = 32. The optimal (in a Bayesian sense) trajectory now spends its first few steps in the valley before ascending to reach the goal location; x = ( 1 , 0 )
Fig. 6 This figure reports the same results as shown in Fig. 4; however, here we have increased the number of time steps from T = 16 to T = 32. The optimal (in a Bayesian sense) trajectory now spends its first few steps in the valley before ascending to reach the goal location; x = ( 1 , 0 )CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 83 references

Research works printed but not linked: 1

  1. Watkins CJ, Dayan P (1992) Q-learning. Mach Learn 8: 279–292

Cited by

See all 264

Metadata

Authors
Friston, K., Samothrakis, S., & Montague, R.
Affiliations
  • Wellcome Centre for Human Neuroimaging (GB)
  • University of Essex (GB)
Corresponding author
Friston, Karl
Venue
Biological Cybernetics
Published
2012-08-03
Volume
106
Issue
8-9
Pages
523-541
ISSN
  • 0340-1200 (ISSN-L)
  • 1432-0770
OpenAlex
W2091405003
Cited by
241
References
83
Citations by year
2026: 11 · 2025: 16 · 2024: 20 · 2023: 24 · 2022: 12 · 2021: 43 · 2020: 30 · 2019: 24 · 2018: 14 · 2017: 12 · 2016: 10 · 2015: 11 · 2014: 7 · 2013: 4 · 2012: 3
FWCI
4.10
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Biological Cybernetics (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • VTechWorks (Virginia Tech) (submittedVersion)
Funders
  • Wellcome Trust (088130)
  • National Institute on Drug Abuse (R01DA011723)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
Keywords
active inference, agency, optimal control, free energy principle, Markov decision process, decision making under uncertainty, Bayesian filtering, prior beliefs, variational inference
MeSH terms
Algorithms, Bayes Theorem, Decision Making, Markov Chains, Models, Neurological

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Active inference and agency: optimal control without cost functions. https://app.cochat.ai/papers/10.1007/s00422-012-0512-8
The paper
Friston, K., Samothrakis, S., & Montague, R. (2012). Active inference and agency: optimal control without cost functions. Biological Cybernetics. https://doi.org/10.1007/s00422-012-0512-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.