CoChat
Sign Up
PreprintCited by 11Reinforcement Learning in RoboticsEmbodied and Extended CognitionAI-based Problem Solving and Planning

The DOI matches its DataCite record; 11 of the 11 references it cites were found in a registry or an index.

Exploration and preference satisfaction trade-off in reward-free learning

Open access
Sajid, Noor, Tigas, Panagiotis, Zakharov, Alexey, Fountas, Zafeirios, Friston, Karl·2021·arXiv
DOI 10.48550/arxiv.2106.04316arXiv 2106.04316
Open this paper in CoChat — free accountOpen PDF

Abstract

Biological agents have meaningful interactions with their environment despite the absence of immediate reward signals. In such instances, the agent can learn preferred modes of behaviour that lead to predictable states -- necessary for survival. In this paper, we pursue the notion that this learnt behaviour can be a consequence of reward-free preference learning that ensures an appropriate trade-off between exploration and preference satisfaction. For this, we introduce a model-based Bayesian agent equipped with a preference learning mechanism (pepper) using conjugate priors. These conjugate priors are used to augment the expected free energy planner for learning preferences over states (or outcomes) across time. Importantly, our approach enables the agent to learn preferences that encourage adaptive behaviour at test time. We illustrate this in the OpenAI Gym FrozenLake and the 3D mini-world environments -- with and without volatility. Given a constant environment, these agents learn confident (i.e., precise) preferences and act to satisfy them. Conversely, in a volatile setting, perpetual preference uncertainty maintains exploratory behaviour. Our experiments suggest that learnable (reward-free) preferences entail a trade-off between exploration and preference satisfaction. Pepper offers a straightforward framework suitable for designing adaptive agents when reward functions cannot be predefined as in real environments.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2106.04316

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The approach proposed by Sajid et al. 2021 could not explicitly integrate extrinsic and intrinsic motivations.

    “This is the approach proposed by (Sajid et al. 2021), but it could not explicitly integrate extrinsic and intrinsic motivations.”

    From Emergence of Goal‐Directed Behaviors via Active Inference With Self‐Prior (2026) · cites this paper as Sajid et al. 2021

    The Introduction (p. 1) identifies preference satisfaction with extrinsic motivation and exploration with intrinsic motivation, then says the approach arbitrates between them. Related work (p. 3) likewise says Pepper trades off exploration and preference satisfaction. The limitation in Concluding remarks (p. 10) is narrower: it does not specify an extrinsic reward function, reducing a designer’s control over behaviour. (LLM, conf=0.84)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 51

  1. [1][1] Achiam, J. and Sastry, S. (2017). Surprise-based intrinsic motivation for deep reinforcement learning. ArXiv , abs/1703.01732. - · arXiv
  2. [3][3] Ball, P., Parker-Holder, J., Pacchiano, A., Choromanski, K., and Roberts, S. (2020). Ready policy one: World building through active learning. In International Conference on Machine Learning , pages 591-601. PMLR. -
  3. [4][4] Barto, A. G. (2013). Intrinsic motivation and reinforcement learning. In Intrinsically motivated learning in natural and artificial systems , pages 17-47. Springer. -
  4. [5][5] Barto, A. G., Singh, S., and Chentanez, N. (2004). Intrinsically motivated learning of hierarchical collections of skills. In Proceedings of the 3rd International Conference on Development and Learning , pages 112-19. Piscataway, NJ. -
  5. [7][7] Blumberg, H. (1920). Hausdorff's grundzüge der mengenlehre. Bulletin of the American Mathematical Society , 27(3):116-129. -
  6. [8][8] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym. arXiv preprint arXiv:1606.01540 . - · arXiv
  7. [10][10] Çatal, O., Verbelen, T., Nauta, J., De Boom, C., and Dhoedt, B. (2020). Learning perception and planning with deep active inference. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3952-3956. IEEE. -
  8. [12][12] Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 . - · arXiv
  9. [13][13] Da Costa, L., Parr, T., Sajid, N., Veselic, S., Neacsu, V., and Friston, K. (2020). Active inference on discrete state-spaces: a synthesis. Journal of Mathematical Psychology , 99:102447. -
  10. [14][14] Fountas, Z., Sajid, N., Mediano, P. A., and Friston, K. (2020). Deep active inference agents using monte-carlo methods. arXiv preprint arXiv:2006.04176 . - · arXiv
  11. [15][15] Friston, K. (2013). Life as we know it. Journal of the Royal Society Interface , 10(86):20130475. -
  12. [19][19] Guo, Z. D., Azar, M. G., Piot, B., Pires, B. A., Pohlen, T., and Munos, R. (2018). Neural predictive belief representations. ArXiv , abs/1811.06407. - · arXiv
  13. [20][20] Haarnoja, T., Ha, S., Zhou, A., Tan, J., Tucker, G., and Levine, S. (2018a). Learning to walk via deep reinforcement learning. arXiv preprint arXiv:1812.11103 . - · arXiv
  14. [21][21] Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018b). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML . -
  15. [22][22] Hadsell, R., Chopra, S., and LeCun, Y. (2006). Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) , volume 2, pages 1735-1742. IEEE. -
  16. [23][23] Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. (2019a). Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603 . - · arXiv
  17. [24][24] Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. (2019b). Learning latent dynamics for planning from pixels. In International Conference on Machine Learning , pages 2555-2565. PMLR. -
  18. [25][25] Hafner, D., Lillicrap, T., Norouzi, M., and Ba, J. (2020). Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193 . - · arXiv
  19. [26][26] Hebb, D. O. (1949). The organization of behavior; a neuropsycholocigal theory. A Wiley Book in Clinical Psychology , 62:78. -
  20. [27][27] Hinton, G. E., Dayan, P., Frey, B. J., and Neal, R. M. (1995). The" wake-sleep" algorithm for unsupervised neural networks. Science , 268(5214):1158-1161. -
  21. [28][28] Hobson, J. A. and Friston, K. J. (2012). Waking and dreaming consciousness: neurobiological and functional considerations. Progress in neurobiology , 98(1):82-98. -
  22. [29][29] Hohwy, J. (2016). The self-evidencing brain. Noûs , 50(2):259-285. -
  23. [30][30] Houthooft, R., Chen, X., Duan, Y., Schulman, J., Turck, F. D., and Abbeel, P. (2016). Vime: Variational information maximizing exploration. In NIPS . -
  24. [31][31] Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu, K. (2016). Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397 . - · arXiv
  25. [32][32] Jin, C., Krishnamurthy, A., Simchowitz, M., and Yu, T. (2020). Reward-free exploration for reinforcement learning. In International Conference on Machine Learning , pages 4870-4879. PMLR. -
  26. [33][33] Kaplan, R. and Friston, K. J. (2018). Planning and navigation as active inference. Biological cybernetics , 112(4):323-343. -
  27. [35][35] Kipf, T., van der Pol, E., and Welling, M. (2020). Contrastive learning of structured world models. ArXiv , abs/1911.12247. - · arXiv
  28. [36][36] Klyubin, A. S., Polani, D., and Nehaniv, C. L. (2005). Empowerment: A universal agent-centric measure of control. In 2005 IEEE Congress on Evolutionary Computation , volume 1, pages 128-135. IEEE. -
  29. [37][37] Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2016). Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint arXiv:1612.01474 . - · arXiv
  30. [38][38] Lee, L., Eysenbach, B., Parisotto, E., Xing, E. P., Levine, S., and Salakhutdinov, R. (2019). Efficient exploration via state marginal matching. ArXiv , abs/1906.05274. - · arXiv
  31. [40][40] Millidge, B., Tschantz, A., and Buckley, C. L. (2021). Whence the Expected Free Energy? Neural Computation , 33(2):447-482. -
  32. [41][41] Misra, I. and Maaten, L. v. d. (2020). Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6707-6717. -
  33. [42][42] Mohamed, S. and Rezende, D. J. (2015). Variational information maximisation for intrinsically motivated reinforcement learning. In NIPS . -
  34. [44][44] Parr, T. and Friston, K. J. (2019). Generalised free energy and active inference. Biological Cybernetics , 113(5-6):495-513. -
  35. [45][45] Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. (2017). Curiosity-driven exploration by self-supervised prediction. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 488-489. -
  36. [48][48] Sajid, N., Ball, P. J., Parr, T., and Friston, K. J. (2021). Active inference: demystified and compared. Neural Computation , 33(3):674-712. -
  37. [52][52] Schmidhuber, J. (2010). Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions on Autonomous Mental Development , 2(3):230-247. -
  38. [55][55] Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., and Pathak, D. (2020). Planning to explore via self-supervised world models. In International Conference on Machine Learning , pages 8583-8592. PMLR. -
  39. [56][56] Shyam, P., Jaśkowski, W., and Gomez, F. (2019a). Model-based active exploration. In International conference on machine learning , pages 5779-5788. PMLR. -
  40. [57][57] Shyam, P., Jaskowski, W., and Gomez, F. (2019b). Model-based active exploration. In ICML . -
  41. [58][58] Singh, S., Barto, A., and Chentanez, N. (2004). Intrinsically motivated reinforcement learning. In NIPS . -
  42. [60][60] Singh, S., Lewis, R. L., Barto, A. G., and Sorg, J. (2010). Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development , 2(2):70-82. -
  43. [61][61] Stadie, B. C., Levine, S., and Abbeel, P. (2015). Incentivizing exploration in reinforcement learning with deep predictive models. NIPS . -
  44. [62][62] Standish, R. K. (2003). Open-ended artificial evolution. International Journal of Computational Intelligence and Applications , 3(02):167-175. -
  45. [65][65] Stooke, A., Lee, K., Abbeel, P., and Laskin, M. (2020). Decoupling representation learning from reinforcement learning. ArXiv , abs/2009.08319. - · arXiv
  46. [66][66] Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction . MIT press. -
  47. [67][67] Tononi, G. and Cirelli, C. (2006). Sleep function and synaptic homeostasis. Sleep medicine reviews , 10(1):49-62. -
  48. [68][68] Tschantz, A., Seth, A. K., and Buckley, C. L. (2020). Learning action-oriented models through active inference. PLoS computational biology , 16(4):e1007805. -
  49. [70][70] van den Oord, A., Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding. ArXiv , abs/1807.03748. - · arXiv
  50. [71][71] Wang, R., Du, S. S., Yang, L. F., and Salakhutdinov, R. (2020). On reward-free reinforcement learning with linear function approximation. arXiv preprint arXiv:2006.11274 . - · arXiv
  51. [72][72] Wilson, R. C., Geana, A., White, J. M., Ludvig, E. A., and Cohen, J. D. (2014). Humans use directed and random exploration to solve the explore-exploit dilemma. Journal of Experimental Psychology: General , 143(6):2074.

Cited by

Sorted by citations

Metadata

Authors
Sajid, N., Tigas, P., Zakharov, A., Fountas, Z., & Friston, K.
Affiliations
  • University College London (GB)
  • University of Oxford (GB)
  • Imperial College London (GB)
  • Huawei Technologies (China) (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2106.04316
arXiv
2106.04316
OpenAlex
W3168182227
Cited by
11
References
11
Citations by year
2024: 4 · 2023: 2 · 2022: 5
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
reward-free learning, preference learning, free energy principle, conjugate priors, preference satisfaction, adaptive agents

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review