Learning action-oriented models through active inference
Open accessPublished version of a preprint
The DOI matches its Crossref record; 83 of the 84 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Converging theories suggest that organisms learn and exploit probabilistic models of their environment. However, it remains unclear how such models can be learned in practice. The open-ended complexity of natural environments means that it is generally infeasible for organisms to model their environment comprehensively. Alternatively, action-oriented models attempt to encode a parsimonious representation of adaptive agent-environment interactions. One approach to learning action-oriented models is to learn online in the presence of goal-directed behaviours. This constrains an agent to behaviourally relevant trajectories, reducing the diversity of the data a model need account for. Unfortunately, this approach can cause models to prematurely converge to sub-optimal solutions, through a process we refer to as a bad-bootstrap. Here, we exploit the normative framework of active inference to show that efficient action-oriented models can be learned by balancing goal-oriented and epistemic (information-seeking) behaviours in a principled manner. We illustrate our approach using a simple agent-based model of bacterial chemotaxis. We first demonstrate that learning via goal-directed behaviour indeed constrains models to behaviorally relevant aspects of the environment, but that this approach is prone to sub-optimal convergence. We then demonstrate that epistemic behaviours facilitate the construction of accurate and comprehensive models, but that these models are not tailored to any specific behavioural niche and are therefore less efficient in their use of data. Finally, we show that active inference agents learn models that are parsimonious, tailored to action, and which avoid bad bootstraps and sub-optimal convergence. Critically, our results indicate that models learned through active inference can support adaptive behaviour in spite of, and indeed because of, their departure from veridical representations of the environment. Our approach provides a principled method for learning adaptive models from limited interactions with an environment, highlighting a route to sample efficient learning algorithms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This unique formulation of confirmation bias as an epistemic phenomenon helps explain how individuals continuously forage their environment for information, but may become stuck in a so-called ‘bad bootstrap’ that simply reinforces existing beliefs about the world, which in the face of new information may lead to sub-optimal behaviour [131].”
From Epistemic Communities under Active Inference (2022) · cites this paper as [131]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Ecological Approach to Visual Perception: Classic Edition
Gibson, J. J. (2014) · doi:10.4324/9781315740218 · Cited by 12,132
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · cited 1×
“Active inference is a normative theory that unifies perception, action and learning under a single imperative—the minimization of variational free energy [42, 43].”
BackgroundMarkov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Puterman, M. L. (1994) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316887 · Cited by 6,024
Rao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483 · cited 2×
“This casts perception as a process of approximate inference, connecting active inference to influential theories such as the Bayesian brain hypothesis (Knill and Pouget, 2004; Gregory, 1980) and predictive coding (Rao and Ballard, 1999).”
MethodBengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
The Bayesian brain: The role of uncertainty in neural coding and computation
Knill, D. C., & Pouget, A. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.10.007 · Cited by 2,588
Chemotaxis in Escherichia coli analysed by Three-dimensional Tracking
BERG, H. C., & BROWN, D. A. (1972) · Nature · doi:10.1038/239500a0 · Cited by 2,231
Research works printed but not linked: 2
- Kuvayev L, Sutton RS. Model-Based Reinforcement Learning with an Approximate, Learned Model. In: in Proceedings of the Ninth Yale Workshop on Adaptive and Learning Systems; 1996. p. 101–105.
- Ha D, Schmidhuber J. World Models. arXiv:180310122. 2018;.
Cited by
Items 101–120 of 120
Beyond Pitch Expectation: Rhythm and Harmony
Pearce, M. T. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780198848004.003.0004
Emotional and Aesthetic Experience
Pearce, M. T. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780198848004.003.0005
Predictive Processing of Melody
Pearce, M. T. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780198848004.003.0003
Pearce, M. T. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780198848004.003.0007
Pearce, M. T. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780198848004.003.0001
Pearce, M. T. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780198848004.003.0006
Pearce, M. T. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780198848004.003.0002
Pearce, M. T. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780198848004.003.0009
Musical Training and Expertise
Pearce, M. T. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780198848004.003.0008
Pearce, M. T. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780198848004.002.0005
Pearce, M. T. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780198848004.002.0003
Working Memory and Self-Directed Inner Speech Enhance Multitask Generalization in Active Inference
Queißer, J. F., & Tani, J. (2025) · Neural Computation · doi:10.1162/neco.a.36
Zenin, M. S., Devyaterikov, A. P., & Palyanov, A. Y. (2025) · Vavilov Journal of Genetics and Breeding · doi:10.18699/vjgb-25-110
Nagatsuka, K., Kutsuzawa, K., Owaki, D., & Hayashibe, M. (2025) · Biomimetics · doi:10.3390/biomimetics11010001
“Most studies have focused on action planning in discrete-time settings [6,18].”
Neurophysiological and computational bases of goal-directed behavior
Basanisi, R. (2021) · theses.fr (ABES) · doi:10.70675/2b7d6591z8d6ez4ea4zbaefz320275b736b1
Yamashita, Y. (2026) · Journal of the Robotics Society of Japan · doi:10.7210/jrsj.44.480
Predictive brains in hyperreal worlds: how XR exploits the indifference to being
Wood, G. M. D. O., & Kober, S. E. (2026) · AI & Society · doi:10.1007/s00146-026-03197-1
“…hyperreal systems can satisfy and sustain SEEKING by continuously offering novelty, coherence (Albarracin et al. 2022), and apparent informational gain, while simultaneously preventing the deeper reflective processes (Tschantz et al. 2020) required to distinguish assumption from judgment.”
Coming to grips with the application of predictive processing in sports and sports coaching
Kennedy, A. (2026) · International Journal of Sports Science & Coaching · doi:10.1177/17479541261466493
Conative inference as a mechanism of allostatic resilience: wanting to be and being to want
Krupnik, V. (2026) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-026-01485-3
Strange attractors: purpose, synchronicity, flow, and the neurodynamics of meaning
Kotler, S., Michael, M., Friston, K. J., & Kelso, S. (2026) · Neuroscience of Consciousness · doi:10.1093/nc/niag063
“This framing echoes Tschantz et al. (2020), who describe “epistemic foraging” as a form of action-oriented model compression, in which agents update internal models by seeking maximal information gain.”
Metadata
- Authors
- Tschantz, A., Seth, A. K., & Buckley, C. L.
- Affiliations
- University of Sussex (GB)
- Canadian Institute for Advanced Research (CA)
- Corresponding author
- Tschantz, Alexander
- Buckley, Christopher L.
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-04-23
- Volume
- 16
- Issue
- 4
- Pages
- e1007805
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC7200021
- OpenAlex
- W3017990858
- Semantic Scholar
- 6417d1bfc8f7dc8df7b04e50f8eb9188784b776a
- Cited by
- 121
- References
- 80
- Citations by year
- 2026: 8 · 2025: 25 · 2024: 9 · 2023: 9 · 2022: 23 · 2021: 31 · 2020: 16
- FWCI
- 10.61
- Type
- journal article
- Language
- en
- Update notices
- new_version (2020-05-05)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Canadian Institute for Advanced Research
- Biotechnology and Biological Sciences Research Council (BB/P022197/1)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Keywords
- active inference, Bayesian learning, goal-directed behavior, information seeking, model convergence, bacterial chemotaxis, sample efficiency, agent-environment interaction
- MeSH terms
- Algorithms, Chemotaxis, Environment, Goals, Learning, Models, Biological, Bacterial Physiological Phenomena, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.