Combining Active Inference and Hierarchical Predictive Coding: A Tutorial Introduction and Case Study
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
This paper combines the active inference formulation of action (Friston, 2009) with hierarchical predictive coding models (Friston, 2003) to provide a proof-of-concept implementation of an active inference agent able to solve a common reinforcement learning baseline -- the cart-pole environment in OpenAI gym. It demonstrates empirically that predictive coding and active inference approaches can be successfully scaled up to tasks more challenging than the mountain car (Friston 2009, 2012). We show that hierarchical predictive coding models can be learned from scratch during the task, and can successfully drive action selection via active inference. To our knowledge, it is the first implemented active inference agent to combine active inference with a hierarchical predictive coding perceptual model. We also provide a tutorial walk-through of the free-energy principle, hierarchical predictive coding, and active inference, including an in-depth derivation of our agent.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120
Rao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483
The Bayesian brain: The role of uncertainty in neural coding and computation
Knill, D. C., & Pouget, A. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.10.007 · Cited by 2,588
Canonical Microcircuits for Predictive Coding
Bastos, A. M., Usrey, W. M., Adams, R. A., Mangun, G. R., Fries, P., & Friston, K. J. (2012) · Neuron · doi:10.1016/j.neuron.2012.10.038 · Cited by 2,421
The free-energy principle: a rough guide to the brain?
Friston, K. (2009) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2009.04.005 · Cited by 1,831
A free energy principle for the brain
Friston, K., Kilner, J., & Harrison, L. (2006) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2006.10.001 · Cited by 1,369
Active inference: a process theory
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., & Pezzulo, G. (2017) · Neural Computation · doi:10.1162/neco_a_00912 · Cited by 1,156
Radical embodiment: neural dynamics and consciousness
Thompson, E., & Varela, F. J. (2001) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01750-2 · Cited by 1,144
Hierarchical Models in the Brain
Friston, K. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000211 · Cited by 952
Variational free energy and the Laplace approximation
Friston, K., Mattout, J., Trujillo-Barreto, N., Ashburner, J., & Penny, W. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2006.08.035 · Cited by 880
Action and behavior: a free-energy formulation
Friston, K. J., Daunizeau, J., Kilner, J., & Kiebel, S. J. (2010) · Biological Cybernetics · doi:10.1007/s00422-010-0364-z · Cited by 866
An embodied cognitive science?
Clark, A. (1999) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(99)01361-3 · Cited by 781
Active inference and epistemic value
Friston, K., Rigoli, F., Ognibene, D., Mathys, C., Fitzgerald, T., & Pezzulo, G. (2015) · Cognitive Neuroscience · doi:10.1080/17588928.2015.1020053 · Cited by 731
Learning and inference in the brain
Friston, K. (2003) · Neural Networks · doi:10.1016/j.neunet.2003.06.005 · Cited by 724
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., ODoherty, J., & Pezzulo, G. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.06.022 · Cited by 674
Action understanding and active inference
Friston, K., Mattout, J., & Kilner, J. (2011) · Biological Cybernetics · doi:10.1007/s00422-011-0424-z · Cited by 665
An aberrant precision account of autism
Lawson, R. P., Rees, G., & Friston, K. J. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00302 · Cited by 641
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016) · arXiv · doi:10.48550/arxiv.1606.01540 · Cited by 582
The Markov blankets of life: autonomy, active inference and the free energy principle
Kirchhoff, M., Parr, T., Palacios, E., Friston, K., & Kiverstein, J. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0792 · Cited by 473
Reinforcement learning or active inference?
Friston, K. J., Daunizeau, J., & Kiebel, S. J. (2009) · PLoS ONE · doi:10.1371/journal.pone.0006421 · Cited by 443
Cerebral hierarchies: predictive processing, precision and the pulvinar
Kanai, R., Komura, Y., Shipp, S., & Friston, K. (2015) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2014.0169 · Cited by 432
A free energy principle for biological systems
Karl, F. (2012) · Entropy · doi:10.3390/e14112100 · Cited by 377
A tutorial on the free-energy framework for modelling perception and learning
Bogacz, R. (2017) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.11.003 · Cited by 283
Active inference and agency: optimal control without cost functions
Friston, K., Samothrakis, S., & Montague, R. (2012) · Biological Cybernetics · doi:10.1007/s00422-012-0512-8 · Cited by 241
Knowing one's place: a free-energy approach to pattern regulation
Friston, K., Levin, M., Sengupta, B., & Pezzulo, G. (2015) · Journal of The Royal Society Interface · doi:10.1098/rsif.2014.1383 · Cited by 234
Friston, K., Stephan, K., Li, B., & Daunizeau, J. (2010) · Mathematical Problems in Engineering · doi:10.1155/2010/621670 · Cited by 171
Active Inference, Attention, and Motor Preparation
Brown, H., Friston, K., & Bestmann, S. (2011) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00218 · Cited by 165
An Active Inference view of cognitive control
Pezzulo, G. (2012) · Frontiers in Psychology · doi:10.3389/fpsyg.2012.00478 · Cited by 68
Knowing when to stop: aberrant precision and evidence accumulation in schizophrenia
Limongi, R., Bohaterewicz, B., Nowicka, M., Plewka, A., & Friston, K. J. (2018) · Schizophrenia Research · doi:10.1016/j.schres.2017.12.018 · Cited by 38
Cited by
A step-by-step tutorial on active inference and its application to empirical data
Smith, R., Friston, K. J., & Whyte, C. J. (2022) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2021.102632 · Cited by 246
Deep active inference as variational policy gradients
Millidge, B. (2020) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2020.102348 · Cited by 115
Self-citationPredictive Coding: a Theoretical and Experimental Review
Millidge, B., Seth, A., & Buckley, C. L. (2021) · arXiv · doi:10.48550/arxiv.2107.12979 · Cited by 106
“Such networks have also been used in the context of reinforcement learning for 1-step environment prediction to enable simple planning and action selection (Millidge, 2019a).”
InfluentialSelf-citationPredictive Coding Approximates Backprop Along Arbitrary Computation Graphs
Millidge, B., Tschantz, A., & Buckley, C. L. (2022) · Neural Computation · doi:10.1162/neco_a_01497 · Cited by 95
Self-citationDeconstructing deep active inference
Champion, T., Grześ, M., Bonheme, L., & Bowman, H. (2023) · arXiv · doi:10.48550/arxiv.2303.01618 · Cited by 7
Metadata
- Authors
- Millidge, B.
- Publisher
- Center for Open Science
- Published
- 2019-03-11
- OpenAlex
- W3161961853
- Cited by
- 22
- References
- 30
- Type
- posted content
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.