Equivalence of Backpropagation and Contrastive Hebbian Learning in a Layered Network
The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
Backpropagation and contrastive Hebbian learning are two methods of training networks with hidden neurons. Backpropagation computes an error signal for the output neurons and spreads it over the hidden neurons. Contrastive Hebbian learning involves clamping the output neurons at desired values and letting the effect spread through feedback connections over the entire network. To investigate the relationship between these two forms of learning, we consider a special case in which they are identical: a multilayer perceptron with linear output units, to which weak feedback connections have been added. In this case, the change in network state caused by clamping the output neurons turns out to be the same as the error signal spread by backpropagation, except for a scalar prefactor. This suggests that the functionality of backpropagation can be realized alternatively by a Hebbian-type learning algorithm, which is suitable for implementation in biological networks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Hopfield, J. J. (1984) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.81.10.3088 · Cited by 6,918
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402
O'Reilly, R. C. (1996) · Neural Computation · doi:10.1162/neco.1996.8.5.895 · Cited by 352 · cited 1×
“LETTER Communicated by Geoffrey Hinton Equivalence of Backpropagation and Contrastive Hebbian Learning in a Layered Network Xiaohui Xie xhx@ai.mit.edu Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, U.S.A.”
Cited by
Deep Learning: Methods and Applications
Deng, L., & Yu, D. (2014) · Foundations and Trends® in Signal Processing · doi:10.1561/2000000039 · Cited by 3,480
“How can computational neuroscience models about hierarchical brain structure help improve engineering deep learning architectures? How may the biologically feasible learning styles in the brain [158, 395] help design more effective and more robust deep learning algorithms? All these issues and those discussed earlier…”
BackgroundDeep Learning: Methods and Applications
Deng, L. (2014) · now publishers, Inc. eBooks · doi:10.1561/9781601988157 · Cited by 1,759
Lillicrap, T. P., Santoro, A., Marris, L., Akerman, C. J., & Hinton, G. (2020) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-0277-3 · Cited by 908
Random synaptic feedback weights support error backpropagation for deep learning
Lillicrap, T. P., Cownden, D., Tweed, D. B., & Akerman, C. J. (2016) · Nature Communications · doi:10.1038/ncomms13276 · Cited by 812
Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation
Scellier, B., & Bengio, Y. (2017) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2017.00024 · Cited by 477
Artificial Neural Networks for Neuroscientists: A Primer
Yang, G. R., & Wang, X.-J. (2020) · Neuron · doi:10.1016/j.neuron.2020.09.005 · Cited by 376
Short-term memory for serial order: A recurrent neural network model.
Botvinick, M. M., & Plaut, D. C. (2006) · Psychological Review · doi:10.1037/0033-295x.113.2.201 · Cited by 369
Stocco, A., Lebiere, C., & Anderson, J. R. (2010) · Psychological Review · doi:10.1037/a0019077 · Cited by 350
Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits
Payeur, A., Guerguiev, J., Zenke, F., Richards, B. A., & Naud, R. (2021) · Nature Neuroscience · doi:10.1038/s41593-021-00857-x · Cited by 292
Towards Biologically Plausible Deep Learning
Bengio, Y., Lee, D.-H., Bornschein, J., Mesnard, T., & Lin, Z. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.04156 · Cited by 276
Direct Feedback Alignment Provides Learning in Deep Neural Networks
Nøkland, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.01596 · Cited by 275
The timing of action determines reward prediction signals in identified midbrain dopamine neurons
Coddington, L. T., & Dudman, J. T. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0245-7 · Cited by 204
Deep neural networks in computational neuroscience
Kietzmann, T. C., McClure, P., & Kriegeskorte, N. (2019) · Oxford Research Encyclopedia of Neuroscience · doi:10.1093/acrefore/9780190264086.013.46 · Cited by 186
Sparse bursts optimize information transmission in a multiplexed neural code
Naud, R., & Sprekeler, H. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1720995115 · Cited by 172
“This neural implementation of multiplexing is distinct from time-division (23, 72) and frequency-division multiplexing (18) and is specific to communication with spike trains.”
Single-cell transcriptomic evidence for dense intracortical neuropeptide networks
Smith, S. J., Sümbül, U., Graybuck, L. T., Collman, F., Seshamani, S., Gala, R., Gliko, O., Elabbady, L., Miller, J. A., Bakken, T. E., et al. (2019) · eLife · doi:10.7554/elife.47889 · Cited by 156
Toward an integration of deep learning and neuroscience
Marblestone, A. H., Wayne, G., & Kording, K. P. (2016) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2016.00094 · Cited by 119
“These include Generalized Recirculation (O’Reilly, 1996), Contrastive Hebbian Learning (Xie and Seung, 2003), random feedback weights together with synaptic homeostasis (Lillicrap et al., 2014; Liao et al., 2015), spike timing dependent plasticity (STDP) with iterative inference and target propagation (Bengio et al.,…”
MethodActivity-difference training of deep neural networks using memristor crossbars
Yi, S.-I., Kendall, J. D., Williams, R. S., & Kumar, S. (2022) · Nature Electronics · doi:10.1038/s41928-022-00869-w · Cited by 75
Deep Predictive Learning: A Comprehensive Model of Three Visual Streams
O'Reilly, R. C., Wyatte, D. R., & Rohrlich, J. (2017) · arXiv · doi:10.48550/arxiv.1709.04654 · Cited by 26
Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks
Salvatori, T., Song, Y., Lukasiewicz, T., Bogacz, R., & Xu, Z. (2021) · arXiv · doi:10.48550/arxiv.2103.03725 · Cited by 16
“But to date, none of the earlier or recent models has bridged the gaps at a degree of showing an equivalence to BP, though some of them [13, 17, 18, 19] show that they approximate (or are equivalent to) BP, but under unrealistic restrictions, e.g., the feedback is sufficiently weak [31, 29, 16].”
BackgroundA predictive-coding network that is both discriminative and generative
Sun, W., & Orchard, J. (2020) · Neural Computation · doi:10.1162/neco_a_01311 · Cited by 11
“The methods of contrastive Hebbian learning (CHL) (O’Reilly, 1996; Xie & Seung, 2003) and equilibrium propagation (Scellier & Bengio, 2017) derive the error gradients using the difference between two equilibrium states of the network.”
Method
Metadata
- Authors
- Xie, X., & Seung, H. S.
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2003-02-01
- Volume
- 15
- Issue
- 2
- Pages
- 441-454
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2134664762
- Cited by
- 184
- References
- 4
- FWCI
- 1.77
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.