CoChat
Sign Up
Cited by 70Embodied and Extended CognitionReinforcement Learning in RoboticsBayesian Modeling and Causal Inference

pymdp: A Python library for active inference in discrete state spaces

Open access
Heins, Conor, Millidge, Beren, Demekas, Daphne, Klein, Brennan, Friston, Karl, Couzin, Iain D., Tschantz, Alexander·2022·Journal of Open Source Software
DOI 10.21105/joss.04098
Open this paper in CoChat — free accountOpen PDF

Abstract

pymdp offers a suite of robust, tested, and modular routines for simulating active inference agents equipped with partially-observable Markov Decision Process (POMDP) generative models.Mathematically, a POMDP comprises a joint distribution over observations o, hidden states s, control states u and hyperparameters φ: P (o, s, u, φ).This joint distribution further factorizes into a set of categorical and Dirichlet distributions: the likelihoods and priors of the generative model.With pymdp, one can build a generative model using a set of prior and likelihood distributions, initialize an agent, and then link it to an external environment to run active inference processes -all in a few lines of code.The Agent and Env (environment) APIs of pymdp are built according to the standardized framework of OpenAIGym commonly used in reinforcement learning, where an agent and environment object recursively exchange observations and actions over time (Brockman et al., 2016).

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.21105/joss.04098

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • EFE computation is validated against pymdp.

    “EFE computation is validated against pymdp [18] (Appendix K).”

    From Expected Free Energy as Belief-Dependent Utility for rho-POMDPs (2026) · cites this paper as [18]

  • PyMDP and the SPM-DEM toolbox have their primary focus on agent creation.

    “Although the realization of environments is also part of popular packages such as PyMDP [11] and the SPM-DEM toolbox [6], these packages have their primary focus on agent creation.”

    From Reactive Environments for Active Inference Agents with RxEnvironments.jl (2025) · cites this paper as [11]

  • Both implementations (our EFE computation and pymdp) agree on observe-vs-commit decisions across all tested belief states (uniform, intermediate, and confident beliefs on the two-state environments).

    “Both implementations agree on observe-vs-commit decisions across all tested belief states (uniform, intermediate, and confident beliefs on the two-state environments).”

    From Expected Free Energy as Belief-Dependent Utility for rho-POMDPs (2026) · cites this paper as [18]

    The Abstract describes pymdp as a tested library for simulating active-inference agents, but it does not report a comparison with the citing authors’ EFE implementation. Searches for observe-versus-commit decisions and agreement across uniform, intermediate, and confident beliefs found no such result in our copy. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 33

  1. Adams, R. A., Vincent, P., Benrimoh, D., Friston, K. J., & Parr, T. (2021). Everything is connected: Inference and attractors in delusions. Schizophrenia Research. https://doi.org/ 10.1016/j.schres.2021.07.032 · DOI
  2. Baioumy, M., Pezzato, C., Corbato, C. H., Hawes, N., & Ferrari, R. (2022). Towards stochastic fault-tolerant control using precision learning and active inference. Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 681-691. https://doi.org/10.1007/978-3-030-93736-2_48 · DOI
  3. Baltieri, M., & Buckley, C. L. (2019). PID control as a process of active inference with linear generative models. Entropy, 21(3), 257. https://doi.org/10.3390/e21030257 · DOI
  4. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016). Openai gym. https://arxiv.org/abs/1606.
  5. Çatal, O., Verbelen, T., Nauta, J., De Boom, C., & Dhoedt, B. (2020). Learning perception and planning with deep active inference. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 3952-3956. https://doi.org/10.1109/ICASSP40776. 2020.9054364 · DOI
  6. Cox, M., Laar, T. van de, & Vries, B. de. (2019). A factor graph approach to automated design of Bayesian signal processing algorithms. International Journal of Approximate Reasoning, 104, 185-204. https://doi.org/10.1016/j.ijar.2018.11.002 · DOI
  7. Ergul, B., Laar, T. van de, Koudahl, M., Roa-Villescas, M., & Vries, B. de. (2020). Learning where to park. International Workshop on Active Inference, 125-132.
  8. Fountas, Z., Sajid, N., Mediano, P. A. M., & Friston, K. J. (2020). Deep ac- tive inference agents using monte-carlo methods. Advances in Neural Infor- mation Processing Systems. https://proceedings.neurips.cc/paper/2020/hash/ 865dfbde8a344b44095495f3591f7407-Abstract.html
  9. Fox, S. (2021). Active inference: Applicability to different types of social organization explained through reference to industrial engineering and quality management. Entropy, 23(2), 198. https://doi.org/10.3390/e23020198 · DOI
  10. Friston, K. J., Daunizeau, J., & Kiebel, S. J. (2009). Reinforcement learning or active inference? PLoS ONE, 4(7), e6421. https://doi.org/10.1371/journal.pone.0006421 · DOI · Paper page
  11. Friston, K. J., Rigoli, F., Ognibene, D., Mathys, C., Fitzgerald, T., & Pezzulo, G. (2015). Active inference and epistemic value. Cognitive Neuroscience, 6(4), 187-214. https: //doi.org/10.1080/17588928.2015.1020053 · DOI · Paper page
  12. Friston, K. J., Samothrakis, S., & Montague, R. (2012). Active inference and agency: Optimal control without cost functions. Biological Cybernetics, 106(8-9), 523-541. https: //doi.org/10.1007/s00422-012-0512-8 · DOI · Paper page
  13. Heins, C., Millidge, B., Demekas, D., Klein, B., Friston, K., Couzin, I., & Tschantz, A. (2022). Pymdp: A python library for active inference in discrete state spaces. https: //arxiv.org/abs/2201.
  14. Holmes, E., Parr, T., Griffiths, T. D., & Friston, K. J. (2021). Active inference, selective attention, and the cocktail party problem. Neuroscience & Biobehavioral Reviews, 131, 1288-1304. https://doi.org/10.1016/j.neubiorev.2021.09.038 · DOI
  15. Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2), 99-134. https: //doi.org/10.1016/S0004-3702(98)00023-X · DOI · Paper page
  16. Laar, T. van de, Senoz, I., Özçelikkale, A., & Wymeersch, H. (2021). Chance-constrained active inference. Neural Computation, 33(10), 2710-2735. https://doi.org/10.1162/neco_ a_01427 · DOI
  17. Laar, T. W. van de, & Vries, B. de. (2019). Simulating active inference processes by message passing. Frontiers in Robotics and AI, 6, 20. https://doi.org/10.3389/frobt.2019.00020 · DOI
  18. Martínez, E. C., Kim, J. W., Barz, T., & Bournazou, M. N. C. (2021). Probabilistic modeling for optimization of bioreactors using reinforcement learning with active inference. Computer Aided Chemical Engineering, 50, 419-424. https://doi.org/10.1016/B978-0-323-88506-5. 50066-8 · DOI
  19. Millidge, B. (2020). Deep active inference as variational policy gradients. Journal of Mathe- matical Psychology, 96, 102348. https://doi.org/10.1016/j.jmp.2020.102348 · DOI · Paper page
  20. Millidge, B., Tschantz, A., Seth, A. K., & Buckley, C. L. (2020). On the relationship between active inference and control as inference. International Workshop on Active Inference, 3-11. https://doi.org/10.1007/978-3-030-64919-7_1 · DOI
  21. Montague, P. R., Dolan, R. J., Friston, K. J., & Dayan, P. (2012). Computational psychiatry. Trends in Cognitive Sciences, 16(1), 72-80. https://doi.org/10.1016/j.tics.2011.11.018 · DOI · Paper page
  22. Parr, T., Markovic, D., Kiebel, S. J., & Friston, K. J. (2019). Neuronal message passing using mean-field, bethe, and marginal approximations. Scientific Reports, 9 (1), 1-18. https://doi.org/10.1038/s41598-018-38246-3 · DOI · Paper page
  23. Parr, T., Rikhye, R. V., Halassa, M. M., & Friston, K. J. (2020). Prefrontal computation as active inference. Cerebral Cortex, 30(2), 682-695. https://doi.org/10.1093/cercor/bhz118 · DOI
  24. Penny, W. D., Friston, K. J., Ashburner, J. T., Kiebel, S. J., & Nichols, T. E. (2007). Statistical parametric mapping: The analysis of functional brain images. https://doi.org/10.1016/ B978-0-12-372560-8.X5000-1 · DOI
  25. Sajid, N., Ball, P. J., Parr, T., & Friston, K. J. (2021). Active inference: Demystified and compared. Neural Computation, 33(3), 674-712. https://doi.org/10.1162/neco_a_01357 · DOI
  26. Smith, R., Friston, K. J., & Whyte, C. J. (2022). A step-by-step tutorial on active inference and its application to empirical data. Journal of Mathematical Psychology, 107, 102632. https://doi.org/10.1016/j.jmp.2021.102632 · DOI · Paper page
  27. Smith, R., Kirlic, N., Stewart, J. L., Touthang, J., Kuplicki, R., Khalsa, S. S., Feinstein, J., Paulus, M. P., & Aupperle, R. L. (2021). Greater decision uncertainty characterizes a transdiagnostic patient sample during approach-avoidance conflict: A computational modelling approach. Journal of Psychiatry & Neuroscience, 46(1), E74. https://doi.org/ 10.1503/jpn.200032 · DOI
  28. Smith, R., Schwartenbeck, P., Stewart, J. L., Kuplicki, R., Ekhtiari, H., Paulus, M. P., & Tulsa 1000 Investigators. (2020). Imprecise action selection in substance use disorder: Evidence for active learning impairments when solving the explore-exploit dilemma. Drug and Alcohol Dependence, 215, 108208. https://doi.org/10.1016/j.drugalcdep.2020.108208 · DOI
  29. Tison, R., & Poirier, P. (2021). Communication as socially extended active inference: An ecological approach to communicative behavior. Ecological Psychology, 33, 197-235. https://doi.org/10.1080/10407413.2021.1965480 · DOI
  30. Tschantz, A., Baltieri, M., Seth, A. K., & Buckley, C. L. (2020). Scaling active inference. 2020 International Joint Conference on Neural Networks (IJCNN), 1-8. https://doi.org/ 10.1109/IJCNN48605.2020.9207382 · DOI
  31. Tschantz, A., Millidge, B., Seth, A. K., & Buckley, C. L. (2020). Reinforcement learning through active inference. Bridging AI and Cognitive Science at the International Conference on Learning Representations. https://baicsworkshop.github.io/pdf/BAICS_37.pdf
  32. Vanderbroeck, M., Baioumy, M., Lans, D. van der, Rooij, R. de, & Werf, T. van der. (2019). Active inference for robot control: A factor graph approach. Student Undergraduate Research E-Journal!, 5, 1-5.
  33. Wirkuttis, N., & Tani, J. (2021). Leading or following? Dyadic robot imitative interaction using the active inference framework. IEEE Robotics and Automation Letters, 6(3), 6024-6031. https://doi.org/10.1109/LRA.2021.3090015 · DOI

Cited by

Sorted by citations

Authors
Heins, C., Millidge, B., Demekas, D., Klein, B., Friston, K., Couzin, I. D., & Tschantz, A.
Affiliations
  • University of Oxford (GB)
  • Imperial College London (GB)
  • Wellcome Centre for Human Neuroimaging (GB)
ORCID
Venue
Journal of Open Source Software
Publisher
The Open Journal
Published
2022-05-04
Volume
7
Issue
73
Pages
4098
ISSN
2475-9066 (ISSN-L)
DOI
10.21105/joss.04098
OpenAlex
W4221159596
Cited by
70
References
6
Citations by year
2026: 14 · 2025: 11 · 2024: 21 · 2023: 15 · 2022: 8 · 2020: 1
FWCI
7.09
Type
journal article
Language
en
Open access
Licence
Locations
  • The Journal of Open Source Software (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, other-oa)
  • arXiv (Cornell University)
  • Zenodo (CERN European Organization for Nuclear Research)
Funders
  • European Commission (860949, HORIZON2020)
  • Deutsche Forschungsgemeinschaft (Excellence Strategy-EXC 2117-422037984, EXC 2117422037984, 2117_422037984, EXC2117, 422037984)
  • Max-Planck-Gesellschaft
  • Office of Naval Research (N00014-64019-1-2556, N00014)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
Keywords
active inference, partially observable Markov decision processes, Bayesian agents, generative models, discrete state space, OpenAI Gym, Python library, reinforcement learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review