Learning probabilistic neural representations with randomly connected circuits
Open accessPublished version of a preprint
The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Abstract
Significance: We present a theory of neural circuits’ design and function, inspired by the random connectivity of real neural circuits and the mathematical power of random projections. Specifically, we introduce a family of statistical models for large neural population codes, a straightforward neural circuit architecture that would implement these models, and a biologically plausible learning rule for such circuits. The resulting neural architecture suggests a design principle for neural circuit—namely, that they learn to compute the mathematical surprise of their inputs, given past inputs, without an explicit teaching signal. We applied these models to recordings from large neural populations in monkeys’ visual and prefrontal cortices and show them to be highly accurate, efficient, and scalable.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Neural networks and physical systems with emergent collective computational abilities.
Hopfield, J. J. (1982) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.79.8.2554 · Cited by 19,270 · cited 1×
“Top-down approaches suggest biologically plausible circuits that solve particular computational tasks but often rely on explicit “teaching signals” or do not even specify how learning could take place (8–14).”
BackgroundGuide for the Care and Use of Laboratory Animals: Eighth Edition
(2011) · doi:10.17226/12910 · Cited by 16,199
Information theory and statistical mechanics
Jaynes, E. T. (1957) · Physical Review · doi:10.1103/physrev.106.620 · Cited by 12,931
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · cited 1×
“The idea that the brain computes the probability of its inputs is supported by evidence of responses to novel inputs or events (28, 29) and has been explored in different contexts such as the hippocampal (30), olfactory (31), and visual (32) systems, as well as in the role of dopamine (33).”
BEAST 2: a software platform for Bayesian evolutionary analysis
Bouckaert, R., Heled, J., Kühnert, D., Vaughan, T., Wu, C.-H., Xie, D., Suchard, M. A., Rambaut, A., & Drummond, A. J. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003537 · Cited by 6,527
A method for solving the convex programming problem with convergence rate O(1/k^2)
Nesterov, Y. E. (1983) · Доклады Академии наук · Cited by 6,318
Rao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483 · cited 1×
“The idea that the brain computes the probability of its inputs is supported by evidence of responses to novel inputs or events (28, 29) and has been explored in different contexts such as the hippocampal (30), olfactory (31), and visual (32) systems, as well as in the role of dopamine (33).”
BackgroundSparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642
Efficient, Multiple-Range Random Walk Algorithm to Calculate the Density of States
Wang, F., & Landau, D. P. (2001) · Physical Review Letters · doi:10.1103/physrevlett.86.2050 · Cited by 2,896
Faisal, A. A., Selen, L. P. J., & Wolpert, D. M. (2008) · Nature Reviews Neuroscience · doi:10.1038/nrn2258 · Cited by 2,883
The analysis of visual motion: a comparison of neuronal and psychophysical performance
Britten, K., Shadlen, M., Newsome, W., & Movshon, J. (1992) · The Journal of Neuroscience · doi:10.1523/jneurosci.12-12-04745.1992 · Cited by 1,958
Reliability of spike timing in neocortical neurons
Mainen, Z. F., & Sejnowski, T. J. (1995) · Science · doi:10.1126/science.7770778 · Cited by 1,950
Weak pairwise correlations imply strongly correlated network states in a neural population
Schneidman, E., Berry, M. J., Segev, R., & Bialek, W. (2006) · Nature · doi:10.1038/nature04701 · Cited by 1,760
Self-citationUsing goal-driven deep learning models to understand sensory cortex
Yamins, D. L. K., & DiCarlo, J. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4244 · Cited by 1,621
Tsodyks, M. V., & Markram, H. (1997) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.94.2.719 · Cited by 1,603
Spatio-temporal correlations and visual signalling in a complete neuronal population
Pillow, J. W., Shlens, J., Paninski, L., Sher, A., Litke, A. M., Chichilnisky, E. J., & Simoncelli, E. P. (2008) · Nature · doi:10.1038/nature07140 · Cited by 1,311
Synaptic theory of working memory
Mongillo, G., Barak, O., & Tsodyks, M. (2008) · Science · doi:10.1126/science.1150769 · Cited by 1,253 · cited 1×
“Updating the synaptic weights to the output neuron would require a back-propagating signal from the cell body [4] and a mechanism that would allow comparing between the synapses’ current and recent activities [5], short-term memory within cells [6, 7], or more complicated local synaptic computations [8].”
Gibbs, J. W. (2010) · Cambridge University Press eBooks · doi:10.1017/cbo9780511686948 · Cited by 1,249
Generating Coherent Patterns of Activity from Chaotic Neural Networks
Sussillo, D., & Abbott, L. F. (2009) · Neuron · doi:10.1016/j.neuron.2009.07.018 · Cited by 1,092
Computational roles for dopamine in behavioural control
Montague, P. R., Hyman, S. E., & Cohen, J. D. (2004) · Nature · doi:10.1038/nature03015 · Cited by 966
Compressed sensing with coherent and redundant dictionaries
Candès, E. J., Eldar, Y. C., Needell, D., & Randall, P. (2011) · Applied and Computational Harmonic Analysis · doi:10.1016/j.acha.2010.10.002 · Cited by 898
Organization of cell assemblies in the hippocampus
Harris, K. D., Csicsvari, J., Hirase, H., Dragoi, G., & Buzsáki, G. (2003) · Nature · doi:10.1038/nature01834 · Cited by 884
Spontaneous Cortical Activity Reveals Hallmarks of an Optimal Internal Model of the Environment
Berkes, P., Orbán, G., Lengyel, M., & Fiser, J. (2011) · Science · doi:10.1126/science.1195870 · Cited by 782
Neural mechanisms for detecting and remembering novel events
Ranganath, C., & Rainer, G. (2003) · Nature Reviews Neuroscience · doi:10.1038/nrn1052 · Cited by 779 · cited 1×
“The idea that the brain computes the probability of its inputs is supported by evidence of responses to novel inputs or events (28, 29) and has been explored in different contexts such as the hippocampal (30), olfactory (31), and visual (32) systems, as well as in the role of dopamine (33).”
BackgroundAction potential initiation and backpropagation in neurons of the mammalian CNS
Stuart, G., Spruston, N., Sakmann, B., & Häusser, M. (1997) · Trends in Neurosciences · doi:10.1016/s0166-2236(96)10075-8 · Cited by 768
Too many cooks? Intrinsic and synaptic homeostatic mechanisms in cortical circuit refinement
Turrigiano, G. (2011) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-060909-153238 · Cited by 671
A comparison of algorithms for maximum entropy parameter estimation
Malouf, R. (2002) · proceeding of the 6th conference on Natural language learning - COLING-02 · doi:10.3115/1118853.1118871 · Cited by 655
Demixed principal component analysis of neural population data
Kobak, D., Brendel, W., Constantinidis, C., Feierstein, C. E., Kepecs, A., Mainen, Z. F., Qi, X.-L., Romo, R., Uchida, N., & Machens, C. K. (2016) · eLife · doi:10.7554/elife.10989 · Cited by 599
Partitioning neuronal variability
Goris, R. L. T., Movshon, J. A., & Simoncelli, E. P. (2014) · Nature Neuroscience · doi:10.1038/nn.3711 · Cited by 528
Robust neuronal dynamics in premotor cortex during motor planning
Li, N., Daie, K., Svoboda, K., & Druckmann, S. (2016) · Nature · doi:10.1038/nature17643 · Cited by 492
The Structure of Multi-Neuron Firing Patterns in Primate Retina
Shlens, J., Field, G. D., Gauthier, J. L., Grivich, M. I., Petrusca, D., Sher, A., Litke, A. M., & Chichilnisky, E. J. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1282-06.2006 · Cited by 468
A distributional code for value in dopamine-based reinforcement learning
Dabney, W., Kurth-Nelson, Z., Uchida, N., Starkweather, C. K., Hassabis, D., Munos, R., & Botvinick, M. (2020) · Nature · doi:10.1038/s41586-019-1924-6 · Cited by 443
Random convergence of olfactory inputs in the Drosophila mushroom body
Caron, S. J. C., Ruta, V., Abbott, L. F., & Axel, R. (2013) · Nature · doi:10.1038/nature12063 · Cited by 409
Learning in Networks of Cortical Neurons
Shahaf, G., & Marom, S. (2001) · The Journal of Neuroscience · doi:10.1523/jneurosci.21-22-08782.2001 · Cited by 399
Barto, A., Mirolli, M., & Baldassarre, G. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00907 · Cited by 397
Dynamic Analysis of Neural Encoding by Point Process Adaptive Filtering
Eden, U. T., Frank, L. M., Barbieri, R., Solo, V., & Brown, E. N. (2004) · Neural Computation · doi:10.1162/089976604773135069 · Cited by 350
Optimal Degrees of Synaptic Connectivity
Litwin-Kumar, A., Harris, K. D., Axel, R., Sompolinsky, H., & Abbott, L. F. (2017) · Neuron · doi:10.1016/j.neuron.2017.01.030 · Cited by 348
Learning in spiking neural networks by reinforcement of stochastic synaptic transmission
Seung, H. S. (2003) · Neuron · doi:10.1016/s0896-6273(03)00761-x · Cited by 340 · cited 1×
“Unlike the traditional roles of noise in computational learning theory for avoiding local minima (53, 54) or finding robust perturbation-based solutions (10), here it is the central component that actively drives learning.”
Probabilistic Interpretation of Population Codes
Zemel, R. S., Dayan, P., & Pouget, A. (1998) · Neural Computation · doi:10.1162/089976698300017818 · Cited by 333
Subtype-specific plasticity of inhibitory circuits in motor cortex during motor learning
Chen, S. X., Kim, A. N., Peters, A. J., & Komiyama, T. (2015) · Nature Neuroscience · doi:10.1038/nn.4049 · Cited by 321
A sensory source for motor variation
Osborne, L. C., Lisberger, S. G., & Bialek, W. (2005) · Nature · doi:10.1038/nature03961 · Cited by 307
Probabilistic Interpretation of Population Codes
Zemel, R. S., Dayan, P., & Pouget, A. (1999) · Neural Codes and Distributed Representations · doi:10.7551/mitpress/4924.003.0011 · Cited by 305
A Maximum Entropy Model Applied to Spatial and Temporal Correlations from Cortical Networks In Vitro
Tang, A., Jackson, D., Hobbs, J., Chen, W., Smith, J. L., Patel, H., Prieto, A., Petrusca, D., Grivich, M. I., Sher, A., et al. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.3359-07.2008 · Cited by 302
Towards Biologically Plausible Deep Learning
Bengio, Y., Lee, D.-H., Bornschein, J., Mesnard, T., & Lin, Z. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.04156 · Cited by 276
Slow waves, sharp waves, ripples, and REM in sleeping dragons
Shein-Idelson, M., Ondracek, J. M., Liaw, H.-P., Reiter, S., & Laurent, G. (2016) · Science · doi:10.1126/science.aaf3621 · Cited by 274 · cited 1×
“We note that neural activity during sleep has been characterized with a replay of neural activity statistics alongside highly regular oscillations of neural population activity [9, 10], which could generate replayed inputs with periodically added noisy echos.”
MethodGanmor, E., Segev, R., & Schneidman, E. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1019641108 · Cited by 259
Self-citationGanguli, S., & Sompolinsky, H. (2012) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-062111-150410 · Cited by 256
Neural Variability and Sampling-Based Probabilistic Representations in the Visual Cortex
Orbán, G., Berkes, P., Fiser, J., & Lengyel, M. (2016) · Neuron · doi:10.1016/j.neuron.2016.09.038 · Cited by 255
Sparseness and Expansion in Sensory Representations
Babadi, B., & Sompolinsky, H. (2014) · Neuron · doi:10.1016/j.neuron.2014.07.035 · Cited by 253
Searching for Collective Behavior in a Large Network of Sensory Neurons
Tkačik, G., Marre, O., Amodei, D., Schneidman, E., Bialek, W., & Berry, M. J. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003408 · Cited by 249
Self-citationLearning by the Dendritic Prediction of Somatic Spiking
Urbanczik, R., & Senn, W. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.030 · Cited by 230 · cited 1×
“Updating the synaptic weights to the output neuron would require a back-propagating signal from the cell body [4] and a mechanism that would allow comparing between the synapses’ current and recent activities [5], short-term memory within cells [6, 7], or more complicated local synaptic computations [8].”
Jabri, M., & Flower, B. (1992) · IEEE Transactions on Neural Networks · doi:10.1109/72.105429 · Cited by 230
The Sparseness of Mixed Selectivity Neurons Controls the Generalization–Discrimination Trade-Off
Barak, O., Rigotti, M., & Fusi, S. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.2753-12.2013 · Cited by 207
Which computational mechanisms operate in the hippocampus during novelty detection?
Kumaran, D., & Maguire, E. A. (2007) · Hippocampus · doi:10.1002/hipo.20326 · Cited by 196
Cocco, S., Leibler, S., & Monasson, R. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0906705106 · Cited by 193
Sleep and memory in mammals, birds and invertebrates
Vorster, A. P., & Born, J. (2015) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2014.09.020 · Cited by 169
Neural processing as causal inference
Lochmann, T., & Deneve, S. (2011) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2011.05.018 · Cited by 158
Learning in neural networks by reinforcement of irregular spiking
Xie, X. S., & Seung, H. S. (2004) · Physical Review E · doi:10.1103/physreve.69.041909 · Cited by 157
Kiani, R., Cueva, C. J., Reppas, J. B., & Newsome, W. T. (2014) · Current Biology · doi:10.1016/j.cub.2014.05.049 · Cited by 155
Self-citationSpiking neurons can discover predictive features by aggregate-label learning
Gütig, R. (2016) · Science · doi:10.1126/science.aab4113 · Cited by 141
New Method for Parameter Estimation in Probabilistic Models: Minimum Probability Flow
Sohl-Dickstein, J., Battaglino, P. B., & DeWeese, M. R. (2011) · Physical Review Letters · doi:10.1103/physrevlett.107.220601 · Cited by 112
Training neural networks with additive noise in the desired signal
Wang, C., & Príncipe, J. C. (1999) · IEEE Transactions on Neural Networks · doi:10.1109/72.809097 · Cited by 100
Predicting non-linear dynamics by stable local learning in a recurrent spiking neural network
Gilra, A., & Gerstner, W. (2017) · eLife · doi:10.7554/elife.28295 · Cited by 100
Adaptive coding for dynamic sensory inference
Młynarski, W. F., & Hermundstad, A. M. (2018) · eLife · doi:10.7554/elife.32055 · Cited by 91
Modeling Higher-Order Correlations within Cortical Microcolumns
Köster, U., Sohl‐Dickstein, J., Gray, C. M., & Olshausen, B. A. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003684 · Cited by 84
A neural data structure for novelty detection
Dasgupta, S., Sheehan, T. C., Stevens, C. F., & Navlakha, S. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1814448115 · Cited by 71
Aitchison, L., & Lengyel, M. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005186 · Cited by 67 · cited 1×
“Since the mapping from stimuli to neural responses (and back) is probabilistic (1–3) and the spaces of stimuli and responses are exponentially large, neural circuits must be performing a form of statistical inference by generalizing from the previously observed spiking patterns (4–7).”
BackgroundEvidence, information, and surprise
Palm, G. (1981) · Biological Cybernetics · doi:10.1007/bf00335160 · Cited by 50
A Normative Theory of Adaptive Dimensionality Reduction in Neural NetworksPreprint
Pehlevan, C., & Chklovskii, D. B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.09426 · Cited by 50
Visual Response Properties of V1 Neurons Projecting to V2 in Macaque
El-Shamayleh, Y., Kumbhani, R. D., Dhruv, N. T., & Movshon, J. A. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.2753-13.2013 · Cited by 48
Flexible and accurate inference and learning for deep generative modelsPreprint
Vértes, E., & Sahani, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11051 · Cited by 48
Bipartite expander Hopfield networks as self-decoding high-capacity error correcting codes
Chaudhuri, R., & Fiete, I. (2019) · neural information processing systems · Cited by 25
Compressive neural representation of sparse, high-dimensional probabilitiesPreprint
Pitkow, Z. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.1800 · Cited by 5
Research works printed but not linked: 2
- Gibbs J. W., Elementary Principles of Statistical Mechanics (Dover Publications, NY; Reprint Edition, 2014).
- Nesterov Y., A method of solving a convex programming problem with convergence rate O (1/k2). Soviet Mathematics Doklady 27, 372– 376 (1983).
Cited by
Alhamrouni, I., Abdul Kahar, N. H., Salem, M., Swadi, M., Zahroui, Y., Kadhim, D. J., Mohamed, F. A., & Alhuyi Nazari, M. (2024) · Applied Sciences · doi:10.3390/app14146214 · Cited by 217
Sequential and efficient neural-population coding of complex task information
Koay, S. A., Charles, A. S., Thiberge, S. Y., Brody, C. D., & Tank, D. W. (2021) · Neuron · doi:10.1016/j.neuron.2021.10.020 · Cited by 91
A neuroscience-inspired spiking neural network for EEG-based auditory spatial attention detection
Faghihi, F., Cai, S., & Moustafa, A. A. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.05.003 · Cited by 45
Rational thoughts in neural codes
Wu, Z., Kwon, M., Daptardar, S., Schrater, P., & Pitkow, X. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912336117 · Cited by 27
Toward Principles of Brain Network Organization and Function
Kulkarni, S., & Bassett, D. S. (2025) · Annual Review of Biophysics · doi:10.1146/annurev-biophys-030722-110624 · Cited by 36
Metadata
- Authors
- Maoz, O., Tkačik, G., Esteki, M. S., Kiani, R., & Schneidman, E.
- Affiliations
- Weizmann Institute of Science (IL)
- Institute of Science and Technology Austria (AT)
- Center for Neural Science, New York University, New York, NY 10003;
- NYU Langone Health (US)
- Corresponding author
- Kiani, Roozbeh
- Schneidman, Elad
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2020-09-18
- Volume
- 117
- Issue
- 40
- Pages
- 25066-25073
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC7547210
- OpenAlex
- W3087227179
- Cited by
- 44
- References
- 76
- Citations by year
- 2026: 8 · 2025: 5 · 2024: 13 · 2023: 6 · 2022: 4 · 2021: 4 · 2020: 3
- FWCI
- 2.28
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Funders
- EC | FP7 | FP7 Ideas: European Research Council (311238)
- Israel Science Foundation (1629/12)
- HHS | NIH | National Institute of Mental Health (R01MH109180)
- Simons Foundation (542997)
- United States - Israel Binational Science Foundation (2014607)
- Pew Charitable Trusts (0)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural dynamics and brain function
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- spike-based computation, local learning rules, neural noise, monkey visual cortex, prefrontal cortex
- MeSH terms
- Machine Learning, Action Potentials, Algorithms, Brain, Humans, Models, Neurological, Nerve Net, Neuronal Plasticity, Neurons, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.