Deep Active Inference for Partially Observable MDPs
Open accessAbstract
Deep active inference has been proposed as a scalable approach to perception and action that deals with large policy and state spaces. However, current models are limited to fully observable domains. In this paper, we describe a deep active inference model that can learn successful policies directly from high-dimensional sensory inputs. The deep learning architecture optimizes a variant of the expected free energy and encodes the continuous state representation by means of a variational autoencoder. We show, in the OpenAI benchmark, that our approach has comparable or better performance than deep Q-learning, a state-of-the-art deep reinforcement learning algorithm.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
State prediction error and KL divergence over states are different quantities, equal only when the two state densities are Gaussian with identity covariance.
“However, the state prediction error and the KL-divergence over states are two different quantities, which are only equal when the two densities over states are Gaussian distributions with identity covariance matrix.”
From Deconstructing Deep Active Inference: A Contrarian Information Gatherer (2024) · cites this paper as van der Himst and Lanillos (2020)
In §2 (p. 3), the paper distinguishes its implemented state-prediction error—mean-squared error between encoded and predicted states—from KL divergence over state densities. It does not state that the quantities are equal when both densities are Gaussian with identity covariance, or that this is the only condition for equality. (LLM, conf=0.85)
Experimental — accuracy not yet measuredThe distribution predicted by the DAI_POMDP encoder network does not have an identity covariance matrix.
“However, the distribution predicted by the encoder network does not have an identity covariance matrix.”
From Deconstructing Deep Active Inference: A Contrarian Information Gatherer (2024) · cites this paper as van der Himst and Lanillos (2020)
In the DAI_POMDP context, the mean squared error and the KL divergence between state densities are not necessarily equivalent.
“That is, in this context, the MSE and the KL-divergence between the densities over state are not necessarily equivalent.”
From Deconstructing Deep Active Inference: A Contrarian Information Gatherer (2024) · cites this paper as van der Himst and Lanillos (2020)
As a result, the DAI_POMDP agent may not always follow the free energy principle.
“As a result, the DAI P OM DP agent may not always follow the free energy principle.”
From Deconstructing Deep Active Inference: A Contrarian Information Gatherer (2024) · cites this paper as van der Himst and Lanillos (2020)
The Deep Active Inference Model section (pp. 2–4) describes the agent as optimizing variational free energy and expecting to act to minimize expected free energy, using approximations to do so. It does not say whether the POMDP agent sometimes fails to follow the free energy principle; searches for such a limitation found no passage addressing it. (LLM, conf=0.70)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 18
- [1]1. Ueltzh¨ offer, K.: Deep active inference. Biological Cybernetics 112 (6), 547-573 (Oct 2018). https://doi.org/10.1007/s00422-018-0785-7 · DOI · Paper page
- [2]2. Sancaktar, C., Lanillos, P.: End-to-end pixel-based deep active inference for body perception and action. arXiv preprint arXiv:2001.05847 (2019) · arXiv
- [3]3. Millidge, B.: Deep active inference as variational policy gradients. Journal of Mathematical Psychology 96 , 102348 (2020). https://doi.org/10.1016/j.jmp.2020.102348 · DOI · Paper page
- [4]4. Tschantz, A., Baltieri, M., Seth, A.K., Buckley, C.L.: Scaling active inference. arXiv Prepr. arXiv:1911.10601v1 (2019) · arXiv
- [5]5. C ¸ atal, O., Wauthier, S., Verbelen, T., Boom, C.D., Dhoedt, B.: Deep active inference for autonomous robot navigation. arXiv Prepr. arXiv:2003.03220v1 (2020) · arXiv
- [6]6. Fountas, Z., Sajid, N., Mediano, P.A.M., Friston, K.: Deep active inference agents using monte-carlo methods. arXiv Prepr. arXiv:2006.04176v1 (2020) · arXiv
- [7]7. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning. arXiv Prepr. arXiv:1312.5602v1 (2013) · arXiv
- [8]8. Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A.: Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine 34 (6), 26-38 (2017)
- [9]9. Friston, K.J., Daunizeau, J., Kilner, J., Kiebel, S.J.: Action and behavior: a free-energy formulation. Biol Cybern 102 , 227-260 (2010). https://doi.org/https://doi.org/10.1007/s00422-010-0364-z · DOI · Paper page
- [10]10. Friston, K.J.: The free-energy principle: a unified brain theory? Nature 11 , 127-138 (2010). https://doi.org/https://doi.org/10.1038/nrn2787 · DOI · Paper page
- [11]11. Adams, R.A., Shipp, S., Friston, K.J.: Predictions not commands: active inference in the motor system. Brain Struct Funct. 218 (3), 611-643 (2012). https://doi.org/doi: 10.1007/s00429-012-0475-5 · DOI · Paper page
- [12]12. Lanillos, P., Pages, J., Cheng, G.: Robot self/other distinction: active inference meets neural networks learning in a mirror. In: Proceedings of the 24th European Conference on Artificial Intelligence (ECAI). pp. 2410 -2416 (2020). https://doi.org/10.3233/FAIA200372 · DOI
- [13]13. Rood, T., van Gerven, M., Lanillos, P.: A deep active inference model of the rubberhand illusion. arXiv Prepr. arXiv:2008.07408 (2020) · arXiv
- [14]14. Oliver, G., Lanillos, P., Cheng, G.: Active inference body perception and action for humanoid robots. arXiv Prepr. arXiv:1906.03022v3 (2019) · arXiv
- [15]15. Friston, K., Trujillo-Barreto, N., Daunizeau, J.: Dem: a variational treatment of dynamic systems. NeuroImage 41 (3), 849-885 (July 2008). https://doi.org/https://doi.org/10.1016/j.neuroimage.2008.02.054 · DOI · Paper page
- [16]16. Parr, T., Friston, K.J.: Generalised free energy and active inference. Biol Cybern 113 , 495-513 (2019). https://doi.org/10.1007/s00422-019-00805-w · DOI · Paper page
- [17]17. Hausknecht, M., Stone, P.: Deep recurrent q-learning for partially observable mdps. In: AAAI Fall Symposium on Sequential Decision Making for Intelligent Agents (AAAI-SDMIA15) (November 2015)
- [18]18. Friston, K.J., Samothrakis, S., Montague, R.: Active inference and agency: optimal control without cost functions. Biol Cybern 106 , 523-541 (2012). https://doi.org/10.1007/s00422-012-0512-8 · DOI · Paper page
Cited by
Sorted by citations
Deconstructing Deep Active Inference: A Contrarian Information Gatherer
Champion, T., Grześ, M., Bonheme, L., & Bowman, H. (2024) · Neural Computation · doi:10.1162/neco_a_01697 · Cited by 5 · see citing works
- Authors
- van der Himst, O., & Lanillos, P.
- Affiliations
- Radboud University Nijmegen (NL)
- ORCID
- Lanillos, Pablo
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Pages
- 61-71
- ISSN
- 1865-0929 (ISSN-L)
- arXiv
- 2009.03622
- OpenAlex
- W3083918609
- Cited by
- 27
- References
- 0
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 7 · 2023: 4 · 2022: 3 · 2021: 7 · 2020: 1
- FWCI
- 1.43
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Communications in computer and information science (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- partially observable Markov decision processes, expected free energy, variational autoencoder, deep reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
