CoChat

The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.

Learning probabilistic neural representations with randomly connected circuits

Open access
Maoz, Ori, Tkačik, Gašper, Esteki, Mohamad Saleh, Kiani, Roozbeh, Schneidman, Elad · 2020 · Proceedings of the National Academy of Sciences

Published version of a preprint

Abstract

Significance: We present a theory of neural circuits’ design and function, inspired by the random connectivity of real neural circuits and the mathematical power of random projections. Specifically, we introduce a family of statistical models for large neural population codes, a straightforward neural circuit architecture that would implement these models, and a biologically plausible learning rule for such circuits. The resulting neural architecture suggests a design principle for neural circuit—namely, that they learn to compute the mathematical surprise of their inputs, given past inputs, without an explicit teaching signal. We applied these models to recordings from large neural populations in monkeys’ visual and prefrontal cortices and show them to be highly accurate, efficient, and scalable.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.1912804117

Cites

76 of 76 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1. A randomly connected neural network, equivalent to an RP model, that learns to generalize from observed inputs to compute the surprise of novel inputs. (A) Examples of six neural population activity patterns at different time points, recorded from 169 neurons in the monkey prefrontal cortex while performing a visual classification task (plotted locations were chosen at random and do not correspond to actual spatial locations). (B) Architecture of a random feed-forward neural circuit based on spiking neurons that can learn to respond with the surprise of its input patterns, x1…xn. The input neurons are connected to an intermediate layer of neurons, hi, with randomly selected synaptic weights aij, which then project to an output neuron with synaptic weights λi. After learning λi, the membrane potential of the output neuron y(x→) will compute −logp^(x1…xn)−logZ, an unnormalized estimate of the surprise, −logp(x1…xn), of the joint input. Note that the same layer of randomly projecting hidden neurons can be reused to simultaneously compute multiple probabilistic models for different output neurons (light color) (SI Appendix, Fig. S9A). (C) The circuit in B is equivalent to a probabilistic model over randomly weighted cliques of neurons, learned by reweighing their contributions, or the maximum entropy model based on random nonlinear statistics of the input.
Fig. 1. A randomly connected neural network, equivalent to an RP model, that learns to generalize from observed inputs to compute the surprise of novel inputs. (A) Examples of six neural population activity patterns at different time points, recorded from 169 neurons in the monkey prefrontal cortex while performing a visual classification task (plotted locations were chosen at random and do not correspond to actual spatial locations). (B) Architecture of a random feed-forward neural circuit based on spiking neurons that can learn to respond with the surprise of its input patterns, x1…xn. The input neurons are connected to an intermediate layer of neurons, hi, with randomly selected synaptic weights aij, which then project to an output neuron with synaptic weights λi. After learning λi, the membrane potential of the output neuron y(x→) will compute −logp^(x1…xn)−logZ, an unnormalized estimate of the surprise, −logp(x1…xn), of the joint input. Note that the same layer of randomly projecting hidden neurons can be reused to simultaneously compute multiple probabilistic models for different output neurons (light color) (SI Appendix, Fig. S9A). (C) The circuit in B is equivalent to a probabilistic model over randomly weighted cliques of neurons, learned by reweighing their contributions, or the maximum entropy model based on random nonlinear statistics of the input.CC BY-NC-ND · PubMed Central
Fig. 2. RP models accurately predict the probability of population activity patterns. (A) Accuracy of different population models in capturing the frequencies of individual population activity patterns in test data from 70 neurons in the monkey visual cortex (Upper) or 50 neurons from the monkey prefrontal cortex (Lower): we compare likelihood ratio of models and test data for an independent model (Left), pairwise maximum entropy model (Center), and RP model (Right). Gray funnels denote 99% CIs of the likelihood ratio resulting from sampling fluctuations. (B) Probability of observing the simultaneous activation of K neurons (population synchrony) in a population of 178 neurons from the primate visual cortex (Upper) and 169 neurons from the primate prefrontal cortex (Lower) in test data and model predictions.
Fig. 2. RP models accurately predict the probability of population activity patterns. (A) Accuracy of different population models in capturing the frequencies of individual population activity patterns in test data from 70 neurons in the monkey visual cortex (Upper) or 50 neurons from the monkey prefrontal cortex (Lower): we compare likelihood ratio of models and test data for an independent model (Left), pairwise maximum entropy model (Center), and RP model (Right). Gray funnels denote 99% CIs of the likelihood ratio resulting from sampling fluctuations. (B) Probability of observing the simultaneous activation of K neurons (population synchrony) in a population of 178 neurons from the primate visual cortex (Upper) and 169 neurons from the primate prefrontal cortex (Lower) in test data and model predictions.CC BY-NC-ND · PubMed Central
Fig. 3. Scalability, optimal sparseness, and efficiency of the RP models. (A) Expected likelihood of RP models for held-out data of individual population activity patterns of 178 neurons in the visual cortex (SI Appendix, Fig. S2A) as a function of the number of projections used in the model (trained using 100,000 samples). Plotted values are median model performance over random choices of projections and divisions of train/test data (error bars are smaller than the marker size) (SI Appendix, Fig. S2D shows a zoomed-in version). (B) Performance of RP models (expected likelihood normalized to a maximum value of one) with different average indegrees (number of incoming connections) of the intermediate neurons, for the visual cortex (Upper) and for prefrontal cortex (Lower), each trained using 100,000 input activity patterns. Different curves denote different sizes of input populations, as denoted on each curve. (C) Expected likelihood of RP trained on population activity patterns of 100 neurons from the monkey visual cortex (SI Appendix, Fig. S2B) as a function of the number of samples in the training data, for RP models using 1,000 RPs (RP 1K), 10,000 RPs (RP 10K), pairwise, k-pairwise, and independent models.
Fig. 3. Scalability, optimal sparseness, and efficiency of the RP models. (A) Expected likelihood of RP models for held-out data of individual population activity patterns of 178 neurons in the visual cortex (SI Appendix, Fig. S2A) as a function of the number of projections used in the model (trained using 100,000 samples). Plotted values are median model performance over random choices of projections and divisions of train/test data (error bars are smaller than the marker size) (SI Appendix, Fig. S2D shows a zoomed-in version). (B) Performance of RP models (expected likelihood normalized to a maximum value of one) with different average indegrees (number of incoming connections) of the intermediate neurons, for the visual cortex (Upper) and for prefrontal cortex (Lower), each trained using 100,000 input activity patterns. Different curves denote different sizes of input populations, as denoted on each curve. (C) Expected likelihood of RP trained on population activity patterns of 100 neurons from the monkey visual cortex (SI Appendix, Fig. S2B) as a function of the number of samples in the training data, for RP models using 1,000 RPs (RP 1K), 10,000 RPs (RP 10K), pairwise, k-pairwise, and independent models.CC BY-NC-ND · PubMed Central
Fig. 4. A local biologically plausible learning rule (LR) for the RP models based on neural noise gives highly accurate models. (A) An LR that trains a circuit to respond with the surprise of its input activity patterns by comparing the response to the input pattern (foreground) with the response to a weakly noisy echo of the input pattern (background). Each synaptic weight is modified according to the differences in activity in the presynaptic neuron, scaled by the relative membrane potentials of the output neuron. (B) RP model trained with the LR and standard gradient descent (GD) on population activity patterns of 100 neurons by repeatedly presenting epochs of the same 100,000 activity patterns. Bold curves denote average over 10 realizations of learning, each plotted in lighter color. (Top) Mean log likelihood of test data under the model (dashed orange: model trained with standard GD; dashed gray: independent model). (Middle) Mean difference in synaptic weights between the LR and standard GD (dashed orange: average difference across multiple realizations of standard GD). Inset shows final values of the individual synaptic weights when learned with the LR vs. standard GD. (Bottom) Example of six individual synaptic weights as they are modified across training epochs. For clarity, the plot uses a linear scale on the x axis for the first epoch and a logarithmic scale afterward.
Fig. 4. A local biologically plausible learning rule (LR) for the RP models based on neural noise gives highly accurate models. (A) An LR that trains a circuit to respond with the surprise of its input activity patterns by comparing the response to the input pattern (foreground) with the response to a weakly noisy echo of the input pattern (background). Each synaptic weight is modified according to the differences in activity in the presynaptic neuron, scaled by the relative membrane potentials of the output neuron. (B) RP model trained with the LR and standard gradient descent (GD) on population activity patterns of 100 neurons by repeatedly presenting epochs of the same 100,000 activity patterns. Bold curves denote average over 10 realizations of learning, each plotted in lighter color. (Top) Mean log likelihood of test data under the model (dashed orange: model trained with standard GD; dashed gray: independent model). (Middle) Mean difference in synaptic weights between the LR and standard GD (dashed orange: average difference across multiple realizations of standard GD). Inset shows final values of the individual synaptic weights when learned with the LR vs. standard GD. (Bottom) Example of six individual synaptic weights as they are modified across training epochs. For clarity, the plot uses a linear scale on the x axis for the first epoch and a logarithmic scale afterward.CC BY-NC-ND · PubMed Central
Fig. 5. Improved RP models using synaptic pruning and replacement. (A) Expected log likelihood of RP models trained with the local learning rule on population activity patterns of 70 neurons from the monkey visual cortex (similar results for the prefrontal cortex are in SI Appendix, Fig. S8A) while periodically pruning weak synapses and replacing them with new randomly chosen projections. Curves denote the performance of models trained with different total average number of replacements per synapse (low: two; high: eight). (B) Average firing rates and (C) Pearson correlations of intermediate units hi in models trained with the learning rule with (orange) or without (purple) pruning and replacement; arrows denote median values.
Fig. 5. Improved RP models using synaptic pruning and replacement. (A) Expected log likelihood of RP models trained with the local learning rule on population activity patterns of 70 neurons from the monkey visual cortex (similar results for the prefrontal cortex are in SI Appendix, Fig. S8A) while periodically pruning weak synapses and replacing them with new randomly chosen projections. Curves denote the performance of models trained with different total average number of replacements per synapse (low: two; high: eight). (B) Average firing rates and (C) Pearson correlations of intermediate units hi in models trained with the learning rule with (orange) or without (purple) pruning and replacement; arrows denote median values.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 76 references

Research works printed but not linked: 2

  1. J. W. Gibbs, Elementary Principles of Statistical Mechanics (Dover Publications, NY; Reprint Edition, 2014).
  2. Y. Nesterov, A method of solving a convex programming problem with convergence rate O (1/k2). Soviet Mathematics Doklady 27, 372–376 (1983).

Cited by

See all 44

Metadata

Authors
Maoz, O., Tkačik, G., Esteki, M. S., Kiani, R., & Schneidman, E.
Affiliations
  • Weizmann Institute of Science (IL)
  • Institute of Science and Technology Austria (AT)
  • Center for Neural Science, New York University, New York, NY 10003;
  • NYU Langone Health (US)
Corresponding author
  • Kiani, Roozbeh
  • Schneidman, Elad
Venue
Proceedings of the National Academy of Sciences
Published
2020-09-18
Volume
117
Issue
40
Pages
25066-25073
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
OpenAlex
W3087227179
Cited by
44
References
76
Citations by year
2026: 8 · 2025: 5 · 2024: 13 · 2023: 6 · 2022: 4 · 2021: 4 · 2020: 3
FWCI
2.28
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
Funders
  • EC | FP7 | FP7 Ideas: European Research Council (311238)
  • Israel Science Foundation (1629/12)
  • HHS | NIH | National Institute of Mental Health (R01MH109180)
  • Simons Foundation (542997)
  • United States - Israel Binational Science Foundation (2014607)
  • Pew Charitable Trusts (0)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural dynamics and brain function
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
Keywords
spike-based computation, local learning rules, neural noise, monkey visual cortex, prefrontal cortex
MeSH terms
Machine Learning, Action Potentials, Algorithms, Brain, Humans, Models, Neurological, Nerve Net, Neuronal Plasticity, Neurons, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Learning probabilistic neural representations with randomly connected circuits. https://app.cochat.ai/papers/10.1073/pnas.1912804117
The paper
Maoz, O., Tkačik, G., Esteki, M. S., Kiani, R., & Schneidman, E. (2020). Learning probabilistic neural representations with randomly connected circuits. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.1912804117

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.