CoChat
Sign Up
PreprintCited by 64Embodied and Extended CognitionReinforcement Learning in RoboticsBayesian Modeling and Causal Inference

Deep active inference agents using Monte-Carlo methods

Open access
Fountas, Zafeirios, Sajid, Noor, Mediano, Pedro A. M., Friston, Karl·2020·arXiv
DOI 10.48550/arxiv.2006.04176arXiv 2006.04176
Open this paper in CoChat — free accountOpen PDF

Abstract

Active inference is a Bayesian framework for understanding biological intelligence. The underlying theory brings together perception and action under one single imperative: minimizing free energy. However, despite its theoretical utility in explaining intelligence, computational implementations have been restricted to low-dimensional and idealized situations. In this paper, we present a neural architecture for building deep active inference agents operating in complex, continuous state-spaces using multiple forms of Monte-Carlo (MC) sampling. For this, we introduce a number of techniques, novel to active inference. These include: i) selecting free-energy-optimal policies via MC tree search, ii) approximating this optimal policy distribution via a feed-forward `habitual' network, iii) predicting future parameter belief updates using MC dropouts and, finally, iv) optimizing state transition precision (a high-end form of attention). Our approach enables agents to learn environmental dynamics efficiently, while maintaining task performance, in relation to reward-based counterparts. We illustrate this in a new toy environment, based on the dSprites data-set, and demonstrate that active inference agents automatically create disentangled representations that are apt for modeling state transitions. In a more complex Animal-AI environment, our agents (using the same neural architecture) are able to simulate future state transitions and actions (i.e., plan), to evince reward-directed navigation - despite temporary suspension of visual input. These results show that deep active inference - equipped with MC methods - provides a flexible framework to develop biologically-inspired intelligent agents, with applications in both machine learning and cognitive science.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.04176

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The DAI_MC agent solves the dSprites environment.

    “Nonetheless, DAI M C solves the dSprites environment.”

    From Deconstructing Deep Active Inference: A Contrarian Information Gatherer (2024) · cites this paper as Fountas et al (2020)

  • Investigations of the expected free energy commonly used in deep active inference indicate that degenerate behaviour can arise from it in certain situations.

    “our investigations of the expected free energy often used in deep active inference (Fountas et al, 2020), suggests that degenerate behaviour can arise from it, in certain situations.”

    From Deconstructing Deep Active Inference: A Contrarian Information Gatherer (2024) · cites this paper as Fountas et al, 2020

  • This conceptual problem affects current deep active inference methods, including Fountas et al (2020).

    “This is a conceptual issue, associated with current deep active inference approaches, such as Fountas et al (2020).”

    From Deconstructing Deep Active Inference: A Contrarian Information Gatherer (2024) · cites this paper as Fountas et al (2020)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 48

  1. [2][2] Karl J Friston. A free energy principle for a particular physics. arXiv preprint arXiv:1906.10184 , 2019. - · arXiv
  2. [7][7] Lancelot Da Costa, Thomas Parr, Noor Sajid, Sebastijan Veselic, Victorita Neacsu, and Karl Friston. Active inference on discrete state-spaces: A synthesis. arXiv preprint arXiv:2001.07203 , 2020. - · arXiv
  3. [8][8] Thomas Parr and Karl J Friston. Uncertainty, epistemics and active inference. Journal of The Royal Society Interface , 14(136):20170376, 2017. -
  4. [10][10] Noor Sajid, Philip J Ball, and Karl J Friston. Active inference: Demystified and compared. arXiv preprint arXiv:1909.10863 , 2019. - · arXiv
  5. [12][12] Maell Cullen, Ben Davey, Karl J Friston, and Rosalyn J. Moran. Active inference in OpenAI Gym: A paradigm for computational investigations into psychiatric illness. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging , 3(9):809-818, 2018. -
  6. [15][15] Ozan Çatal, Johannes Nauta, Tim Verbelen, Pieter Simoens, and Bart Dhoedt. Bayesian policy selection using active inference. arXiv preprint arXiv:1904.08149 , 2019. - · arXiv
  7. [16][16] Alexander Tschantz, Manuel Baltieri, Anil Seth, Christopher L Buckley, et al. Scaling active inference. arXiv preprint arXiv:1911.10601 , 2019. - · arXiv
  8. [17][17] Beren Millidge. Deep active inference as variational policy gradients. Journal of Mathematical Psychology , 96:102348, 2020. -
  9. [18][18] David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association , 112(518):859-877, 2017. -
  10. [19][19] Thomas Parr and Karl J Friston. Generalised free energy and active inference. Biological Cybernetics , 113(5-6):495-513, 2019. -
  11. [21][21] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction . MIT press, 2018. -
  12. [23][23] Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International Conference on Computers and Games , pages 72-83. Springer, 2006. -
  13. [24][24] Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of Monte Carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in Games , 4(1):1-43, 2012. -
  14. [25][25] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature , 550(7676):354-359, 2017. -
  15. [26][26] Alexandre Piché, Valentin Thomas, Cyril Ibrahim, Yoshua Bengio, and Chris Pal. Probabilistic planning with sequential monte carlo methods. In International Conference on Learning Representations , 2018. -
  16. [27][27] Alexander Tschantz, Beren Millidge, Anil K Seth, and Christopher L Buckley. Control as hybrid inference. arXiv preprint arXiv:2007.05838 , 2020. - · arXiv
  17. [28][28] Joseph Marino and Yisong Yue. An inference perspective on model-based reinforcement learning. ICML Workshop on Generative Modeling and Model-Based Reasoning for Robotics and AI , 2019. -
  18. [29][29] Matthijs Van Der Meer, Zeb Kurth-Nelson, and A David Redish. Information processing in decision-making systems. The Neuroscientist , 18(4):342-359, 2012. -
  19. [31][31] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. International Conference on Learning Representations , 2(5):6, 2017. -
  20. [33][33] Matthew Crosby, Benjamin Beyret, and Marta Halina. The Animal-AI olympics. Nature Machine Intelligence , 1(5):257-257, 2019. -
  21. [35][35] Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. In Advances in Neural Information Processing Systems , pages 517-526, 2018. -
  22. [36][36] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint arXiv:1802.05983 , 2018. - · arXiv
  23. [38][38] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015. -
  24. [39][39] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937, 2016. -
  25. [41][41] Takuya Isomura and Karl J Friston. In vitro neural networks minimise variational free energy. Nature Scientific Reports , 8(1):1-14, 2018. -
  26. [42][42] Rick A Adams, Stewart Shipp, and Karl J Friston. Predictions not commands: Active inference in the motor system. Brain Structure and Function , 218(3):611-643, 2013. -
  27. [43][43] Levente Kocsis and Csaba Szepesvári. Bandit based Monte-Carlo planning. In European Conference on Machine Learning , pages 282-293. Springer, 2006. -
  28. [44][44] Xiaoxiao Guo, Satinder Singh, Honglak Lee, Richard L Lewis, and Xiaoshi Wang. Deep learning for real-time Atari game play using offline Monte-Carlo tree search planning. In Advances in Neural Information Processing Systems , pages 3338-3346, 2014. -
  29. [45][45] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature , 529 (7587):484, 2016. -
  30. [46][46] Ozan Çatal, Tim Verbelen, Johannes Nauta, Cedric De Boom, and Bart Dhoedt. Learning perception and planning with deep active inference. In IEEE International Conference on Acoustics, Speech and Signal Processing , pages 3952-3956, 2020. -
  31. [47][47] Joseph Snider, Dongpyo Lee, Howard Poizner, and Sergei Gepshtein. Prospective optimization with limited resources. PLoS Computational Biology , 11(9), 2015. -
  32. [48][48] Alec Solway and Matthew M Botvinick. Evidence integration in model-based tree search. Proceedings of the National Academy of Sciences , 112(37):11708-11713, 2015. -
  33. [50][50] Dennis H Holding. Counting backward during chess move choice. Bulletin of the Psychonomic Society , 27(5):421-424, 1989. -
  34. [52][52] Bruce D Burns. The effects of speed on skilled chess performance. Psychological Science , 15 (7):442-447, 2004. -
  35. [53][53] Frenk Van Harreveld, Eric-Jan Wagenmakers, and Han LJ Van Der Maas. The effects of time pressure on chess skill: An investigation into fast and slow processes underlying expert performance. Psychological Research , 71(5):591-597, 2007. -
  36. [54][54] Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. arXiv preprint arXiv:1812.02833 , 2018. - · arXiv
  37. [55][55] Minyoung Kim, Yuting Wang, Pritish Sahu, and Vladimir Pavlovic. Bayes-Factor-VAE: Hierarchical Bayesian deep auto-encoder models for factor disentanglement. In IEEE International Conference on Computer Vision , pages 2979-2987, 2019. -
  38. [56][56] Hadi Fatemi Shariatpanahi and Majid Nili Ahmadabadi. Biologically inspired framework for learning and abstract representation of attention control. In Attention in Cognitive Systems. Theories and Systems from an Interdisciplinary Viewpoint , pages 307-324, 2007. -
  39. [57][57] Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo J Rezende. Towards interpretable reinforcement learning using attention augmented agents. In Advances in Neural Information Processing Systems , pages 12329-12338, 2019. -
  40. [59][59] Peter Dayan, Sham Kakade, and Read P Montague. Learning and selective attention. Nature Neuroscience , 3(11):1218-1223, 2000. -
  41. [60][60] Farhan Baluch and Laurent Itti. Mechanisms of top-down attention. Trends in Neurosciences , 34(4):210-224, 2011. -
  42. [61][61] Yuka Sasaki, Jose E Nanez, and Takeo Watanabe. Advances in visual perceptual learning and plasticity. Nature Reviews Neuroscience , 11(1):53-60, 2010. -
  43. [63][63] Peter Dayan and Angela J Yu. ACh, uncertainty, and cortical inference. In Advances in Neural Information Processing Systems , pages 189-196, 2002. -
  44. [64][64] Q Gu. Neuromodulatory transmitter systems in the cortex and their role in cortical plasticity. Neuroscience , 111(4):815-835, 2002. -
  45. [67][67] Thomas Parr. The Computational Neurology of Active Vision . PhD thesis, University College London, 2019. -
  46. [68][68] Matthew Fellows, Anuj Mahajan, Tim GJ Rudner, and Shimon Whiteson. Virel: A variational inference framework for reinforcement learning. In Advances in Neural Information Processing Systems , pages 7122-7136, 2019. -
  47. [69][69] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909 , 2018. - · arXiv
  48. [70][70] Thomas Parr and Karl J Friston. The computational anatomy of visual neglect. Cerebral Cortex , 28(2):777-790, 2018.

Cited by

Sorted by citations

Authors
Fountas, Z., Sajid, N., Mediano, P. A. M., & Friston, K.
Affiliations
  • Wellcome Centre for Human Neuroimaging (GB)
  • University College London (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
33
Pages
11662-11675
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2006.04176
arXiv
2006.04176
OpenAlex
W3033115783
Cited by
64
References
15
Citations by year
2026: 1 · 2025: 5 · 2024: 17 · 2023: 11 · 2022: 15 · 2021: 11 · 2020: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Wellcome Trust (210920, 088130, 210920/Z/18/Z, 088130/Z/09/Z)
  • Medical Research Council (MR/S502522/1)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
Keywords
active inference, free energy principle, Monte Carlo tree search, Monte Carlo dropout, disentangled representations, planning, biological intelligence, deep learning architectures

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review