A theory of local learning, the learning channel, and the optimality of backpropagation
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
In a physical neural system, where storage and processing are intimately intertwined, the rules for adjusting the synaptic weights can only depend on variables that are available locally, such as the activity of the pre- and post-synaptic neurons, resulting in local learning rules. A systematic framework for studying the space of local learning rules is obtained by first specifying the nature of the local variables, and then the functional form that ties them together into each learning rule. Such a framework enables also the systematic discovery of new learning rules and exploration of relationships between learning rules and group symmetries. We study polynomial local learning rules stratified by their degree and analyze their behavior and capabilities in both linear and non-linear units and networks. Stacking local learning rules in deep feedforward networks leads to deep local learning. While deep local learning can learn interesting representations, it cannot learn complex input-output functions, even when targets are available for the top layer. Learning complex input-output functions requires local deep learning where target information is communicated to the deep layers through a backward learning channel. The nature of the communicated information about the targets and the structure of the learning channel partition the space of learning algorithms. For any learning algorithm, the capacity of the learning channel can be defined as the number of bits provided about the error gradient per weight, divided by the number of required operations per weight. We estimate the capacity associated with several learning algorithms and show that backpropagation outperforms them by simultaneously maximizing the information rate and minimizing the computational cost. This result is also shown to be true for recurrent networks, by unfolding them in time. The theory clarifies the concept of Hebbian learning, establishes the power and limitations of local learning rules, introduces the learning channel which enables a formal analysis of the optimality of backpropagation, and explains the sparsity of the space of learning rules discovered so far.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Neural networks and physical systems with emergent collective computational abilities.
Hopfield, J. J. (1982) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.79.8.2554 · Cited by 19,270
The perceptron: A probabilistic model for information storage and organization in the brain.
Rosenblatt, F. (1958) · Psychological Review · doi:10.1037/h0042519 · Cited by 12,280
An Introduction to Statistical Modeling of Extreme Values
Coles, S. (2001) · Springer Series in Statistics · doi:10.1007/978-1-4471-3675-0 · Cited by 7,352
Fukushima, K. (1980) · Biological Cybernetics · doi:10.1007/bf00344251 · Cited by 5,759 · cited 2×
“While less successful than backpropagation in applications, it has periodically inspired the developmen t of theories aimed at capturing the essence of neural learni ng [26, 22, 29].”
BackgroundEmergence of simple-cell receptive field properties by learning a sparse code for natural images
Olshausen, B. A., & Field, D. J. (1996) · Nature · doi:10.1038/381607a0 · Cited by 5,694
Greedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
WIDROW, B., & HOFF, M. E. (1960) · doi:10.21236/ad0241531 · Cited by 4,009
Cited by
Neftci, E. O., Mostafa, H., & Zenke, F. (2019) · IEEE Signal Processing Magazine · doi:10.1109/msp.2019.2931595 · Cited by 1,538
Theories of Error Back-Propagation in the Brain
Whittington, J. C. R., & Bogacz, R. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.12.005 · Cited by 420
“…(e.g., handwritten digit classi fi cation) [12 – 14, The criticism of weight symmetry has been addressed by demonstrating that even if the errors in arti fi cial neural networks are back-propagated by random connections, good performance in classi fi cation tasks can still be achieved [21,23 – 27].”
BackgroundSupervised learning in spiking neural networks: A review of algorithms and evaluations
Wang, X., Lin, X., & Dang, X. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.02.011 · Cited by 277
Deep learning in biomedical data science
Baldi, P. (2018) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-080917-013343 · Cited by 119
Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks
Salvatori, T., Song, Y., Lukasiewicz, T., Bogacz, R., & Xu, Z. (2021) · arXiv · doi:10.48550/arxiv.2103.03725 · Cited by 16
“…since the flourishing of BP, because on the one hand, the underlying principles of BP are unrealistic for an implementation in the brain [30], but on the other hand, BP outperforms all alternative discovered frameworks [3] and closely reproduces activity patterns observed in the cortex [12, 32, 11].”
Background
Metadata
- Authors
- Baldi, P., & Sadowski, P.
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2016-11
- Volume
- 83
- Pages
- 51-74
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W2482290902
- Cited by
- 74
- References
- 19
- FWCI
- 8.85
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A theory of local learning, the learning channel, and the optimality of backpropagation. https://app.cochat.ai/papers/10.1016/j.neunet.2016.07.006
- The paper
- Baldi, P., & Sadowski, P. (2016). A theory of local learning, the learning channel, and the optimality of backpropagation. Neural Networks. https://doi.org/10.1016/j.neunet.2016.07.006
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.