The DOI matches its Crossref record; 66 of the 67 references it cites were found in a registry or an index.
Reinforcement Learning or Active Inference?
Open accessAbstract
This paper questions the need for reinforcement learning or control theory when optimising behaviour. We show that it is fairly simple to teach an agent complicated and adaptive behaviours using a free-energy formulation of perception. In this formulation, agents adjust their internal states and sampling of the environment to minimize their free-energy. Such agents learn causal structure in the environment and sample it in an adaptive and self-supervised fashion. This results in behavioural policies that reproduce those optimised by reinforcement learning and dynamic programming. Critically, we do not need to invoke the notion of reward, value or utility. We illustrate these points by solving a benchmark problem in dynamic programming; namely the mountain-car problem, using active perception or inference under the free-energy principle. The ensuing proof-of-concept may be important because the free-energy formulation furnishes a unified account of both action and perception and may speak to a reappraisal of the role of dopamine in the brain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, this hierarchical elaboration dissolves any apparent mechanistic distinction between the optimization processes that underlie different control or learning schemes, suggesting that they are all manifestations of Active Inference – under various contexts or conditions (Friston et al., 2009).”
From Active Inference, homeostatic regulation and adaptive behavioural control (2015) · cites this paper as Friston et al., 2009
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rescorla, R. A. (1972) · Medical Entomology and Zoology · Cited by 9,462
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Autopoiesis and Cognition: The Realization of the Living
Maturana, H. R., & Varela, F. J. (1980) · Boston Studies in the Philosophy and History of Science · doi:10.1007/978-94-009-8947-4 · Cited by 5,815
Emergence of simple-cell receptive field properties by learning a sparse code for natural images
Olshausen, B. A., & Field, D. J. (1996) · Nature · doi:10.1038/381607a0 · Cited by 5,694
Rao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483
A theory of cortical responses.
Friston, K. (2005) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2005.1622 · Cited by 4,578
Self-citationPredictive reward signal of dopamine neurons
Schultz, W. (1998) · Journal of Neurophysiology · doi:10.1152/jn.1998.80.1.1 · Cited by 4,564
Classical conditioning II: Current research and theory.
Black, A. H., & Prokasy, W. F. (1972) · Appleton-Century-Crofts eBooks · Cited by 4,449
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1023/a:1022676722315 · Cited by 3,907
An internal model for sensorimotor integration
Wolpert, D. M., Ghahramani, Z., & Jordan, M. I. (1995) · Science · doi:10.1126/science.7569931 · Cited by 3,553
Cornfeld, I. P., Sinaǐ, Y. G., & Fomin, S. (1982) · Cited by 2,852
The Bayesian brain: The role of uncertainty in neural coding and computation
Knill, D. C., & Pouget, A. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.10.007 · Cited by 2,588
Cz., P. (1896) · Monatshefte für Mathematik und Physik · doi:10.1007/bf01708548 · Cited by 2,055
Handbuch der physiologischen Optik
Helmholtz, H. L. F. (2011) · DFG Viewer (DFG Viewer) · Cited by 1,947
Uncertainty, Neuromodulation, and Attention
Yu, A. J., & Dayan, P. (2005) · Neuron · doi:10.1016/j.neuron.2005.04.026 · Cited by 1,774 · cited 1×
“In biological formulations of the free-energy principle, current thinking is that dopamine might encode the precision of prior expectations [39,48].”
BackgroundSynaptic depression and cortical gain control
Abbott, L. F., Varela, J. A., Sen, K., & Nelson, S. B. (1997) · Science · doi:10.1126/science.275.5297.221 · Cited by 1,690
Hierarchical Bayesian inference in the visual cortex
Lee, T. S., & Mumford, D. (2003) · Journal of the Optical Society of America A · doi:10.1364/josaa.20.001434 · Cited by 1,473
Toward a modern theory of adaptive networks: Expectation and prediction.
Sutton, R. S., & Barto, A. G. (1981) · Psychological Review · doi:10.1037/0033-295x.88.2.135 · Cited by 1,450 · cited 1×
“In control theory, value acts as a guiding function by establishing gradients, which the agent can ascend [2,3,5].”
BackgroundA free energy principle for the brain
Friston, K., Kilner, J., & Harrison, L. (2006) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2006.10.001 · Cited by 1,369
Self-citationObject perception as Bayesian inference
Kersten, D., Mamassian, P., & Yuille, A. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.142005 · Cited by 1,269
Dayan, P., Hinton, G. E., Neal, R. M., & Zemel, R. S. (1995) · Neural Computation · doi:10.1162/neco.1995.7.5.889 · Cited by 1,207
A computational neuroanatomy for motor control
Shadmehr, R., & Krakauer, J. W. (2008) · Experimental Brain Research · doi:10.1007/s00221-008-1280-5 · Cited by 1,187
On the computational architecture of the neocortex: II The role of cortico-cortical loops
Mumford, D. (1992) · Biological Cybernetics · doi:10.1007/bf00198477 · Cited by 1,074
Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding
Sutton, R. S. (1995) · Cited by 1,072
On the theory of dynamic programming
Bellman, R. (1952) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.38.8.716 · Cited by 998
Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events
Horvitz, J. C. (2000) · Neuroscience · doi:10.1016/s0306-4522(00)00019-1 · Cited by 967
Hierarchical Models in the Brain
Friston, K. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000211 · Cited by 952 · cited 4×
“Although generalised coordinates may appear to complicate matters, they actually simplify inference greatly; see [13] and [39] for details.”
BackgroundMethodInfluentialSelf-citationJONES, J. E. (1971) · Nature · doi:10.1038/233571b0 · Cited by 952
On the computational architecture of the neocortex: I. The role of the thalamo-cortical loop
Mumford, D. (1991) · Biological Cybernetics · doi:10.1007/bf00202389 · Cited by 929
Variational free energy and the Laplace approximation
Friston, K., Mattout, J., Trujillo-Barreto, N., Ashburner, J., & Penny, W. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2006.08.035 · Cited by 880
Self-citationThe short-latency dopamine signal: a role in discovering novel actions?
Redgrave, P., & Gurney, K. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn2022 · Cited by 762
Metalearning and neuromodulation
Doya, K. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00044-8 · Cited by 704
A Hierarchy of Time-Scales and the Brain
Kiebel, S. J., Daunizeau, J., & Friston, K. J. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000209 · Cited by 694 · cited 1×
“The same scheme used in this paper has been used to simulate a range of biological processes; ranging from perceptual categorisation of bird-song [57] to perceptual learning during the mismatch negativity paradigm [10].”
MethodSelf-citationFriston, K. J., & Stephan, K. E. (2007) · Synthese · doi:10.1007/s11229-007-9237-y · Cited by 673
Self-citationDopamine: generalization and bonuses
Kakade, S., & Dayan, P. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00048-5 · Cited by 480
The computational neurobiology of learning and reward
Daw, N. D., & Doya, K. (2006) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2006.03.006 · Cited by 465
Relevance of Error: What Drives Motor Adaptation?
Wei, K., & Körding, K. (2009) · Journal of Neurophysiology · doi:10.1152/jn.90545.2008 · Cited by 422
Ballard, D. H., Hinton, G. E., & Sejnowski, T. J. (1983) · Nature · doi:10.1038/306021a0 · Cited by 356
Linearly-solvable Markov decision problems
Todorov, E. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0176 · Cited by 334
Bayesian Spiking Neurons I: Inference
Deneve, S. (2008) · Neural Computation · doi:10.1162/neco.2008.20.1.91 · Cited by 313
Bee foraging in uncertain environments using predictive hebbian learning
Montague, P. R., Dayan, P., Person, C., & Sejnowski, T. J. (1995) · Nature · doi:10.1038/377725a0 · Cited by 311
Behavioural studies of strategic thinking in games
Camerer, C. F. (2003) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(03)00094-9 · Cited by 304
DE MAQUINAS Y SERES VIVOS, AUTOPOIESIS: la organización de lo vivo "Quinta edición"
Maturana, H. R., & Varela, F. J. (1998) · Cited by 290
DEM: A variational treatment of dynamic systems
Friston, K. J., Trujillo-Barreto, N., & Daunizeau, J. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.02.054 · Cited by 278 · cited 3×
“Although generalised coordinates may appear to complicate matters, they actually simplify inference greatly; see [13] and [39] for details.”
BackgroundSelf-citationValue-dependent selection in the brain: Simulation in a synthetic neural model
Friston, K. J., Tononi, G., Reeke, G. N., Sporns, O., & Edelman, G. M. (1994) · Neuroscience · doi:10.1016/0306-4522(94)90592-4 · Cited by 268
Self-citationPerceptual Neural Organization: Some Approaches Based on Network Models and Information Theory
Linsker, R. (1990) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.13.030190.001353 · Cited by 248
Handbuch der physiologischen Optik
von Helmholtz, H. (n.d.) · Cited by 246
Environmentally mediated synergy between perception and behaviour in mobile robots
Verschure, P. F. M. J., Voegtlin, T., & Douglas, R. J. (2003) · Nature · doi:10.1038/nature02024 · Cited by 216
Hayashi, S., Wakabayashi, K., Ishikawa, A., Nagai, H., Saito, M., Maruyama, M., Takahashi, T., Ozawa, T., Tsuji, S., & Takahashi, H. (2000) · Movement Disorders · doi:10.1002/1531-8257(200009)15:5<884::aid-mds1019>3.0.co;2-8 · Cited by 212
Eye movement statistics in humans are consistent with an optimal search strategy
Najemnik, J., & Geisler, W. S. (2008) · Journal of Vision · doi:10.1167/8.3.4 · Cited by 208
Wörgötter, F., & Porr, B. (2005) · Neural Computation · doi:10.1162/0899766053011555 · Cited by 203
Coevolutionary dynamics in large, but finite populations
Traulsen, A., Claussen, J. C., & Hauert, C. (2006) · Physical Review E · doi:10.1103/physreve.74.011901 · Cited by 172
Tschacher, W., & Haken, H. (2007) · New Ideas in Psychology · doi:10.1016/j.newideapsych.2006.09.002 · Cited by 152
Adaptive, Fast Walking in a Biped Robot under Neuronal Control and Learning
Manoonpong, P., Geng, T., Kulvicius, T., Porr, B., & Wörgötter, F. (2007) · PLoS Computational Biology · doi:10.1371/journal.pcbi.0030134 · Cited by 119
A non-equilibrium free energy theorem for deterministic systems
EVANS, D. J. (2003) · Molecular Physics · doi:10.1080/0026897031000085173 · Cited by 100
Demetrius, L. (2000) · Journal of Theoretical Biology · doi:10.1006/jtbi.2000.2106 · Cited by 93
PATTERN RECOGNITION AND THE RESPONSES OF SENSORY NEURONS*
Barlow, H. B. (1969) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1969.tb14019.x · Cited by 93
Computational models of the basal ganglia
Gillies, A., & Arbuthnott, G. W. (2000) · Movement Disorders · doi:10.1002/1531-8257(200009)15:5<762::aid-mds1002>3.0.co;2-2 · Cited by 87
Free energy minimisation algorithmfor decoding and cryptanalysis
MacKay, D. J. C. (1995) · Electronics Letters · doi:10.1049/el:19950331 · Cited by 79
Verschure, P. F. M. J., & Voegtlin, T. (1998) · Neural Networks · doi:10.1016/s0893-6080(98)00029-x · Cited by 73
Insights from models of rhythmic motor systems
Prinz, A. A. (2006) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2006.10.001 · Cited by 24
Correction: Adaptive, Fast Walking in a Biped Robot under Neuronal Control and Learning
Manoonpong, P., Geng, T., Kulvičius, T., Porr, B., & Wörgötter, F. (2007) · PLoS Computational Biology · doi:10.1371/journal.pcbi.0030191 · Cited by 18
Browning Agents and Active Particles
(2007) · Springer Series in Synergetics · doi:10.1007/978-3-540-73845-9 · Cited by 17
Gontar, V. (2000) · Chaos, Solitons & Fractals · doi:10.1016/s0960-0779(98)00287-2 · Cited by 16
Development of receptive fields in a closed-loop behavioural system
Kulvičius, T., Porr, B., & Wörgötter, F. (2006) · Neurocomputing · doi:10.1016/j.neucom.2006.10.132 · Cited by 3
Research works printed but not linked: 3
- DJC MacKay, 1995, Free-energy minimisation algorithm for decoding and cryptoanalysis., Electronics Letters, 31, 445
- T Ozaki, 1992, A bridge between nonlinear time-series models and nonlinear stochastic dynamical systems: A local linearization approach., Statistica Sin, 2, 113
- ME Tipping, 2001, Sparse Bayesian learning and the Relevance Vector Machine., J. Machine Learning Research, 1, 211
Cited by
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120
Self-citationWhatever next? Predictive brains, situated agents, and the future of cognitive science
Clark, A. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12000477 · Cited by 5,924
The free-energy principle: a rough guide to the brain?
Friston, K. (2009) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2009.04.005 · Cited by 1,831
Self-citationAction and behavior: a free-energy formulation
Friston, K. J., Daunizeau, J., Kilner, J., & Kiebel, S. J. (2010) · Biological Cybernetics · doi:10.1007/s00422-010-0364-z · Cited by 866
“This entails adjusting the parameters to maximize the ensemble density, p(˜ x|m) at the target location, as described in Friston et al. (2009) .”
BackgroundMethodSelf-citationThe Computational Anatomy of Psychosis
Adams, R. A., Stephan, K. E., Brown, H. R., Frith, C. D., & Friston, K. J. (2013) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2013.00047 · Cited by 850
“In short, active inference can be regarded www.frontiersin.org May 2013 | Volume 4 | Article 47 | 9 as equipping a generalized predictive coding scheme with classical reflex arcs: see Friston et al. (2009) and Adams et al. (2013) for details.”
BackgroundSelf-citation
Metadata
- Authors
- Friston, K. J., Daunizeau, J., & Kiebel, S. J.
- Affiliations
- Wellcome Centre for Human Neuroimaging (GB)
- Corresponding author
- Friston, Karl J.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2009-07-29
- Volume
- 4
- Issue
- 7
- Pages
- e6421
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC2713351
- OpenAlex
- W2137411342
- Cited by
- 443
- References
- 67
- Citations by year
- 2026: 26 · 2025: 30 · 2024: 43 · 2023: 37 · 2022: 42 · 2021: 36 · 2020: 45 · 2019: 38 · 2018: 25 · 2017: 19 · 2016: 14 · 2015: 15 · 2014: 20 · 2013: 12 · 2012: 18
- FWCI
- 6.64
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- Max Planck Digital Library (submittedVersion, other-oa)
- MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by)
- Zurich Open Repository and Archive (University of Zurich)
- Funders
- Wellcome Trust (WT056750, 088130)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Arts and Humanities › Philosophy › Free Will and Agency
- Keywords
- free energy principle, active inference, reinforcement learning, active perception, causal discovery, self-supervised learning, dynamic programming, mountain car problem, dopamine, perception and action
- MeSH terms
- Humans, Learning, Models, Theoretical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.