Learning in Neural Networks with Material Synapses
Open accessThe DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
We discuss the long term maintenance of acquired memory in synaptic connections of a perpetually learning electronic device. This is affected by ascribing each synapse a finite number of stable states in which it can maintain for indefinitely long periods. Learning uncorrelated stimuli is expressed as a stochastic process produced by the neural activities on the synapses. In several interesting cases the stochastic process can be analyzed in detail, leading to a clarification of the performance of the network, as an associative memory, during the process of uninterrupted learning. The stochastic nature of the process and the existence of an asymptotic distribution for the synaptic values in the network imply generically that the memory is a palimpsest but capacity is as low as log N for a network of N neurons. The only way we find for avoiding this tight constraint is to allow the parameters governing the learning process (the coding level of the stimuli; the transition probabilities for potentiation and depression and the number of stable synaptic levels) to depend on the number of neurons. It is shown that a network with synapses that have two stable states can dynamically learn with optimal storage efficiency, be a palimpsest, and maintain its (associative) memory for an indefinitely long time provided the coding level is low and depression is equilibrated against potentiation. We suggest that an option so easily implementable in material devices would not have been overlooked by biology. Finally we discuss the stochastic learning on synapses with variable number of stable synaptic states.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Neural networks and physical systems with emergent collective computational abilities.
Hopfield, J. J. (1982) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.79.8.2554 · Cited by 19,270
Non-Holographic Associative Memory
WILLSHAW, D. J., BUNEMAN, O. P., & LONGUET-HIGGINS, H. C. (1969) · Nature · doi:10.1038/222960a0 · Cited by 979 · cited 5×
“where p+ = b/(a + b) is the fraction of synapses with value J+. Note that the Willshaw (1969) model has J+ = 1,J- = 0 q-=O.”
BackgroundMethodResultInfluentialStatistical mechanics of neural networks near saturation
Amit, D. J., Gutfreund, H., & Sompolinsky, H. (1987) · Annals of Physics · doi:10.1016/0003-4916(87)90092-3 · Cited by 849
Self-citationNeuronal correlate of visual associative long-term memory in the primate temporal cortex
Miyashita, Y. (1988) · Nature · doi:10.1038/335817a0 · Cited by 739
Synaptic transmission between individual pyramidal neurons of the rat visual cortex in vitro
Mason, A., Nicoll, A., & Stratford, K. (1991) · The Journal of Neuroscience · doi:10.1523/jneurosci.11-01-00072.1991 · Cited by 429
The Hebbian paradigm reintegrated: local reverberations as internal representations
Amit, D. J. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040164 · Cited by 388
Self-citationStorage capacity of memory networks with binary couplings
Krauth, W., & Mézard, M. (1989) · Journal de Physique · doi:10.1051/jphys:0198900500200305700 · Cited by 276
Conversion of Temporal Correlations Between Stimuli to Spatial Correlations Between Attractors
Griniasty, M., Tsodyks, M. V., & Amit, D. J. (1993) · Neural Computation · doi:10.1162/neco.1993.5.1.1 · Cited by 223
Self-citationNetworks of Formal Neurons and Memory Palimpsests
Nadal, J. P., Toulouse, G., Changeux, J. P., & Dehaene, S. (1986) · Europhysics Letters (EPL) · doi:10.1209/0295-5075/1/10/008 · Cited by 212
Parisi, G. (1986) · Journal of Physics A: Mathematical and General · doi:10.1088/0305-4470/19/10/011 · Cited by 165
Capacity of neural networks with discrete synaptic couplings
Gutfreund, H., & Stein, Y. (1990) · Journal of Physics A Mathematical and General · doi:10.1088/0305-4470/23/12/036 · Cited by 87
Constraints on learning in dynamic synapses
Amit, D. J., & Fusi, S. (1992) · Network Computation in Neural Systems · doi:10.1088/0954-898x_3_4_008 · Cited by 59
Self-citationASSOCIATIVE MEMORY IN NEURAL NETWORKS WITH BINARY SYNAPSES
Tsodyks, M. V. (1990) · Modern Physics Letters B · doi:10.1142/s0217984990000891 · Cited by 55
The Hebbian paradigm reintegrated: Local reverberations as internal representations
Freeman, W. J. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0004022x · Cited by 36
Constraints on learning in dynamic synapses
Amit, D. J., & Fusi, S. (1992) · Network Computation in Neural Systems · doi:10.1088/0954-898x/3/4/008 · Cited by 23
Self-citationAdequate input for learning in attractor neural networks
Amit, D. J., & Brunel, N. (1993) · Network Computation in Neural Systems · doi:10.1088/0954-898x_4_2_003 · Cited by 10
Self-citationAdequate Input for Learning in Attractor Neural Networks
Amit, D. J., & Brunel, N. (1993) · Springer eBooks · doi:10.1007/978-1-4471-2063-6_6 · Cited by 8
Self-citationReverberations of Hebbian thinking
Rauschecker, J. P. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040358 · Cited by 6
Not the module does memory make – but the network
Fuster, J. M. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040231 · Cited by 6
Another ANN model for the Miyashita experiments
Morita, M. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040310 · Cited by 5
Mathematics of Hebbian attractors
Hirsch, M. W. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040243 · Cited by 5
How to decide whether a neural representation is a cognitive concept?
Raijmakers, M. E. J., & Molenaar, P. C. M. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040346 · Cited by 5
Distributed cell assemblies and detailed cell models
Lansner, A., & Fransén, E. A. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040292 · Cited by 4
The problems of cognitive dynamical models
Petitot, J. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040322 · Cited by 4
Local or transcortical assemblies? Some evidence from cognitive neuroscience
Pulvermüller, F., & Preissl, H. T. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040334 · Cited by 4
Association and computation with cell assemblies
van der Velde, F. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0004036x · Cited by 4
Reverberation reconsidered: On the path to cognitive theory
Chown, E. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0004019x · Cited by 4
Additional tests of Amit's attractor neural networks
Hoffman, R. E. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040255 · Cited by 4
The functional meaning of reverberations for sensoric and contextual encoding
Klimesch, W. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040279 · Cited by 4
An evolutionary perspective on Hebb's reverberatory representations
Krakauer, D. C., & Houston, A. I. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040280 · Cited by 4
Hebb's accomplishments misunderstood
Hucka, M., Weaver, M., & Kaplan, S. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040267 · Cited by 4
Dalenoort, G. J., & de Vries, P. H. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040206 · Cited by 3
Empirical and theoretical active memory: The proper context
Amit, D. J. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040383 · Cited by 3
Self-citationAre single-cell data sufficient for testing neural network models?
Ahissar, E. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040176 · Cited by 3
Attractors – don't get sucked in
Milner, P. M. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040309 · Cited by 3
Bienenstock, E., & Geman, S. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040188 · Cited by 3
How representation works is more important than what representations are
Edelman, S. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040218 · Cited by 3
How do local reverberations achieve global integration?
Wright, J. J. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040371 · Cited by 2
Cited by
Amit, D. (1997) · Cerebral Cortex · doi:10.1093/cercor/7.3.237 · Cited by 1,059
Self-citationIndiveri, G., Chicca, E., & Douglas, R. (2006) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2005.860850 · Cited by 1,009
Cascade Models of Synaptically Stored Memories
Fusi, S., Drew, P. J., & Abbott, L. F. (2005) · Neuron · doi:10.1016/j.neuron.2005.02.001 · Cited by 569
“With this level of noise and Sig- anced against each other (Amit and Fusi, 1994; Fusi, 2002).”
BackgroundMethodResultInfluentialSelf-citationNeuromorphic Electronic Circuits for Building Autonomous Cognitive Systems
Chicca, E., Stefanini, F., Bartolozzi, C., & Indiveri, G. (2014) · Proceedings of the IEEE · doi:10.1109/jproc.2014.2313954 · Cited by 552
The Hebbian paradigm reintegrated: local reverberations as internal representations
Amit, D. J. (1995) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00040164 · Cited by 388
Self-citation
Metadata
- Authors
- Amit, D. J., & Fusi, S.
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 1994-09
- Volume
- 6
- Issue
- 5
- Pages
- 957-982
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2101222204
- Cited by
- 290
- References
- 38
- FWCI
- 17.08
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.