When Response Variability Increases Neural Network Robustness to Synaptic Noise
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Cortical sensory neurons are known to be highly variable, in the sense that responses evoked by identical stimuli often change dramatically from trial to trial. The origin of this variability is uncertain, but it is usually interpreted as detrimental noise that reduces the computational accuracy of neural circuits. Here we investigate the possibility that such response variability might in fact be beneficial, because it may partially compensate for a decrease in accuracy due to stochastic changes in the synaptic strengths of a network. We study the interplay between two kinds of noise, response (or neuronal) noise and synaptic noise, by analyzing their joint influence on the accuracy of neural networks trained to perform various tasks. We find an interesting, generic interaction: when fluctuations in the synaptic connections are proportional to their strengths (multiplicative noise), a certain amount of response noise in the input neurons can significantly improve network performance, compared to the same network without response noise. Performance is enhanced because response noise and multiplicative synaptic noise are in some ways equivalent. So if the algorithm used to find the optimal synaptic weights can take into account the variability of the model neurons, it can also take into account the variability of the synapses. Thus, the connection patterns generated with response noise are typically more resistant to synaptic degradation than those obtained without response noise. As a consequence of this interplay, if multiplicative synaptic noise is present, it is better to have response noise in the network than not to have it. These results are demonstrated analytically for the most basic network consisting of two input neurons and one output neuron performing a simple classification task, but computer simulations show that the phenomenon persists in a wide range of architectures, including recurrent (attractor) networks and sensorimotor networks that perform coordinate transformations. The results suggest that response variability could play an important dynamic role in networks that continuously learn.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gammaitoni, L., Hänggi, P., Jung, P., & Marchesoni, F. (1998) · Reviews of Modern Physics · doi:10.1103/revmodphys.70.223 · Cited by 5,443
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770
Softky, W., & Koch, C. (1993) · The Journal of Neuroscience · doi:10.1523/jneurosci.13-01-00334.1993 · Cited by 1,689 · cited 4×
“…is highly variable, as measured either by the temporal structure of spike trains produced during constant stimulation conditions, or by spike counts collected in a given time interval and compared across identical behavioral trials (Dean, 1981; Softky and Koch, 1992, 1993; Holt et al., 1996).”
BackgroundResultInfluentialCorrelated neuronal activity and the flow of neural information
Salinas, E., & Sejnowski, T. J. (2001) · Nature Reviews Neuroscience · doi:10.1038/35086012 · Cited by 1,333
Self-citationTraining with Noise is Equivalent to Tikhonov Regularization
Bishop, C. M. (1995) · Neural Computation · doi:10.1162/neco.1995.7.1.108 · Cited by 1,325
Compte, A. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.9.910 · Cited by 1,143
Zhang, K. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-06-02112.1996 · Cited by 1,001 · cited 1×
“Such stable ‘bump’-shaped activity is observed in various neural models, including those for cortical hypercolumns (Hansel and Sompolinsky, 1998), headdirection cells (Zhang, 1996; Laing and Chow, 2001) and working memory circuits (Compte et al., 2000).”
BackgroundSalinas, E., & Thier, P. (2000) · Neuron · doi:10.1016/s0896-6273(00)00004-0 · Cited by 400 · cited 2×
“The amplitude — but not the selectivity — of a visually-triggered response, represented by fi(x), depends on the direction of gaze (Andersen et al., 1985; Brotchie et al., 1995; Salinas and Thier, 2000).”
BackgroundSelf-citationLearning to see: experience and attention in primary visual cortex
Crist, R. E., Li, W., & Gilbert, C. D. (2001) · Nature Neuroscience · doi:10.1038/87470 · Cited by 364 · cited 1×
“In the cortex, this problem must be quite significant, given that even primary sensory areas show a large capacity for reorganization (Wang et al., 1995; Kilgard and Merzenich, 1998; Crist et al., 2001).”
BackgroundStability of the memory of eye position in a recurrent network of conductance-based model neurons
Seung, H. S., Lee, D. D., Reis, B. Y., & Tank, D. W. (2000) · Neuron · doi:10.1016/s0896-6273(00)81155-1 · Cited by 339
Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training
Murray, A. F., & Edwards, P. J. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.317730 · Cited by 201 · cited 2×
“Also, adding noise to the synaptic connections of a network during learning produces networks that, after training, are more robust to synaptic corruption and have a higher capacity to generalize (Murray and Edwards, 1994).”
Background
Cited by
The cognitive neuroscience of ageing
Grady, C. (2012) · Nature Reviews Neuroscience · doi:10.1038/nrn3256 · Cited by 1,286
Moment-to-moment brain signal variability: A next frontier in human brain mapping?
Garrett, D. D., Samanez-Larkin, G. R., MacDonald, S. W. S., Lindenberger, U., McIntosh, A. R., & Grady, C. L. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.02.015 · Cited by 585
Increased Brain Signal Variability Accompanies Lower Behavioral Variability in Development
McIntosh, A. R., Kovacevic, N., & Itier, R. J. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000106 · Cited by 431
The Modulation of BOLD Variability between Cognitive States Varies by Age and Processing Speed
Garrett, D. D., Kovacevic, N., McIntosh, A. R., & Grady, C. L. (2013) · Cerebral Cortex · doi:10.1093/cercor/bhs055 · Cited by 276
Karalunas, S. L., Geurts, H. M., Konrad, K., Bender, S., & Nigg, J. T. (2014) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12217 · Cited by 278
Metadata
- Authors
- Basalyga, G., & Salinas, E.
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2006-06
- Volume
- 18
- Issue
- 6
- Pages
- 1349-1379
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W1996063134
- Cited by
- 56
- References
- 11
- FWCI
- 1.53
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.