CoChat
Sign Up
PreprintCited by 4Reinforcement Learning in RoboticsDomain Adaptation and Few-Shot LearningNeural Networks and Applications

The DOI matches its DataCite record.

Divergent representations of ethological visual inputs emerge from supervised, unsupervised, and reinforcement learning

Open access
Lindsay, Grace W., Merel, Josh, Mrsic-Flogel, Tom, Sahani, Maneesh·2021·arXiv
DOI 10.48550/arxiv.2112.02027arXiv 2112.02027
Open this paper in CoChat — free accountOpen PDF

Abstract

Artificial neural systems trained using reinforcement, supervised, and unsupervised learning all acquire internal representations of high dimensional input. To what extent these representations depend on the different learning objectives is largely unknown. Here we compare the representations learned by eight different convolutional neural networks, each with identical ResNet architectures and trained on the same family of egocentric images, but embedded within different learning systems. Specifically, the representations are trained to guide action in a compound reinforcement learning task; to predict one or a combination of three task-related targets with supervision; or using one of three different unsupervised objectives. Using representational similarity analysis, we find that the network trained with reinforcement learning differs most from the other networks. Using metrics inspired by the neuroscience literature, we find that the model trained with reinforcement learning has a sparse and high-dimensional representation wherein individual images are represented with very different patterns of neural activity. Further analysis suggests these representations may arise in order to guide long-term behavior and goal-seeking in the RL agent. Finally, we compare the representations learned by the RL agent to neural activity from mouse visual cortex and find it to perform as well or better than other models. Our results provide insights into how the properties of neural representations are influenced by objective functions and can inform transfer learning approaches.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2112.02027

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Training under the RL paradigm introduces greater variance in results; two agent architectures trained under RL might achieve drastically different final performances or learned representations.

    “Moreover, training under the RL paradigm introduces greater variance in the results. Namely, two agent architectures trained under RL might achieve drastically different final performances or learned representations (Lindsay et al., 2021).”

    From Design and evaluation of a global workspace agent embodied in a realistic multimodal environment (2024) · cites this paper as Lindsay et al., 2021

    The Abstract (p. 1) and Discussion (p. 12) compare one RL-trained network with networks trained under other objectives; they do not compare two RL-trained architectures. The Model Performance section (p. 6) reports performance differences among supervised and unsupervised models, not variance in final performance or representations across RL agents. Searches for RL-run variability and different RL architectures found no passage addressing the claim. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 42

  1. Baktash Babadi and Haim Sompolinsky. Sparseness and expansion in sensory representations. Neuron , 83(5):1213-1226, 2014.
  2. Shahab Bakhtiari, Patrick Mineault, Timothy Lillicrap, Christopher Pack, and Blake Richards. The functional specialization of visual cortex emerges from training parallel pathways with selfsupervised predictive learning. Advances in Neural Information Processing Systems , 34, 2021.
  3. Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Proceedings of ICML workshop on unsupervised and transfer learning , pp. 17-36. JMLR Workshop and Conference Proceedings, 2012.
  4. Saskia EJ de Vries, Jerome A Lecoq, Michael A Buice, Peter A Groblewski, Gabriel K Ocker, Michael Oliver, David Feng, Nicholas Cain, Peter Ledochowitsch, Daniel Millman, et al. A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex. Nature neuroscience , 23(1):138-151, 2020.
  5. Peter F¨ oldiak. Forming sparse representations by local anti-hebbian learning. Biological cybernetics , 64(2):165-170, 1990.
  6. Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. A theory of multineuronal dimensionality, dynamics and measurement. BioRxiv , pp. 214262, 2017.
  7. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
  8. Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning , volume 1. MIT Press, 2016.
  9. Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio G´ omez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. Rl unplugged: A collection of benchmarks for offline reinforcement learning. Advances in Neural Information Processing Systems , 33, 2020.
  10. Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L McClelland, and Adam Santoro. Environmental drivers of systematicity and generalization in a situated agent. In International Conference on Learning Representations , 2019.
  11. Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference on Learning Representations , 2017.
  12. Tim C Kietzmann, Patrick McClure, and Nikolaus Kriegeskorte. Deep neural networks in computational neuroscience. In Oxford research encyclopedia of neuroscience . Oxford, 2019.
  13. Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013. · arXiv
  14. Talia Konkle and George A Alvarez. Instance-level contrastive learning yields human brain-like representation without category-supervision. bioRxiv , 2020.
  15. Nikolaus Kriegeskorte and Xue-Xin Wei. Neural tuning and representational geometry. arXiv preprint arXiv:2104.09743 , 2021. · arXiv
  16. Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning , pp. 56395650. PMLR, 2020.
  17. Alessandro Lazaric. Transfer in reinforcement learning: a framework and a survey. In Reinforcement Learning , pp. 143-173. Springer, 2012.
  18. Lucas Lehnert, Michael L Littman, and Michael J Frank. Reward-predictive representations generalize across tasks in reinforcement learning. PLoS computational biology , 16(10):e1008317, 2020.
  19. William Lotter, Gabriel Kreiman, and David Cox. A neural network trained for prediction mimics diverse features of biological neurons and perception. Nature Machine Intelligence , 2(4):210219, 2020.
  20. Josh Merel, Diego Aldarondo, Jesse Marshall, Yuval Tassa, Greg Wayne, and Bence Olveczky. Deep neuroethology of a virtual rodent. In International Conference on Learning Representations , 2020.
  21. Aran Nayebi, Nathan CL Kong, Chengxu Zhuang, Justin L Gardner, Anthony M Norcia, and Daniel LK Yamins. Unsupervised models of mouse visual cortex. bioRxiv , 2021.
  22. Hamed Nili, Cai Wingfield, Alexander Walther, Li Su, William Marslen-Wilson, and Nikolaus Kriegeskorte. A toolbox for representational similarity analysis. PLoS computational biology , 10(4):e1003553, 2014.
  23. Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature , 381(6583):607-609, 1996.
  24. Bruno A Olshausen and David J Field. Sparse coding of sensory inputs. Current opinion in neurobiology , 14(4):481-487, 2004.
  25. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018. · arXiv
  26. Eddie J Rafols, Mark B Ring, Richard S Sutton, and Brian Tanner. Using predictive representations to improve generalization in reinforcement learning. In IJCAI , pp. 835-840, 2005.
  27. Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience , 22(11):1761-1770, 2019.
  28. Mattia Rigotti, Omri Barak, Melissa R Warden, Xiao-Jing Wang, Nathaniel D Daw, Earl K Miller, and Stefano Fusi. The importance of mixed selectivity in complex cognitive tasks. Nature , 497 (7451):585-590, 2013.
  29. Edmund T Rolls and Alessandro Treves. The relative advantages of sparse versus distributed encoding for associative neuronal networks in the brain. Network: computation in neural systems , 1(4): 407, 1990.
  30. Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In International Conference on Learning Representations , 2021.
  31. Joshua H Siegle, Xiaoxuan Jia, S´ everine Durand, Sam Gale, Corbett Bennett, Nile Graddis, Greggory Heller, Tamina K Ramirez, Hannah Choi, Jennifer A Luviano, et al. Survey of spiking in the mouse visual system reveals functional hierarchy. Nature , 592(7852):86-92, 2021.
  32. Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning , pp. 9870-9879. PMLR, 2021.
  33. Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research , 10(7), 2009.
  34. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pp. 5026-5033. IEEE, 2012.
  35. William E Vinje and Jack L Gallant. Sparse coding and decorrelation in primary visual cortex during natural vision. Science , 287(5456):1273-1276, 2000.
  36. Aria Wang, Michael Tarr, and Leila Wehbe. Neural taskonomy: Inferring the similarity of taskderived representations from brain activity. Advances in Neural Information Processing Systems , 32:15501-15511, 2019.
  37. Greg Wayne, Chia-Chun Hung, David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka GrabskaBarwinska, Jack Rae, Piotr Mirowski, Joel Z Leibo, Adam Santoro, et al. Unsupervised predictive memory in a goal-directed agent. arXiv preprint arXiv:1803.10760 , 2018. · arXiv
  38. Benjamin Willmore and David J Tolhurst. Characterizing the sparseness of neural codes. Network: Computation in Neural Systems , 12(3):255, 2001.
  39. Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience , 19(3):356-365, 2016.
  40. Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential decision making. arXiv preprint arXiv:2102.05815 , 2021. · arXiv
  41. Amir R. Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2018.
  42. Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C Frank, James J DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences , 118(3), 2021.

Cited by

Sorted by citations

Related

Metadata

Authors
Lindsay, G. W., Merel, J., Mrsic-Flogel, T., & Sahani, M.
Affiliations
Sainsbury Wellcome Centre (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2112.02027
arXiv
2112.02027
OpenAlex
W4226209460
Cited by
4
References
0
Citations by year
2025: 1 · 2024: 1 · 2023: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-sa/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
neural representations, reinforcement learning, supervised learning, unsupervised learning, convolutional neural networks, representational similarity analysis, egocentric vision, visual cortex, mouse visual cortex, sparse representation, goal-directed behavior, transfer learning, learning objectives

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review