Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks
Open accessAbstract
Predictive coding networks (PCNs) are an influential model for information processing in the brain. They have appealing theoretical interpretations and offer a single mechanism that accounts for diverse perceptual phenomena of the brain. On the other hand, backpropagation (BP) is commonly regarded to be the most successful learning method in modern machine learning. Thus, it is exciting that recent work formulates inference learning (IL) that trains PCNs to approximate BP. However, there are several remaining critical issues: (i) IL is an approximation to BP with unrealistic/non-trivial requirements, (ii) IL approximates BP in single-step weight updates; whether it leads to the same point as BP after the weight updates are conducted for more steps is unknown, and (iii) IL is computationally significantly more costly than BP. To solve these issues, a variant of IL that is strictly equivalent to BP in fully connected networks has been proposed. In this work, we build on this result by showing that it also holds for more complex architectures, namely, convolutional neural networks and (many-to-one) recurrent neural networks. To our knowledge, we are the first to show that a biologically plausible algorithm is able to exactly replicate the accuracy of BP on such complex architectures, bridging the existing gap between IL and BP, and setting an unprecedented performance for PCNs, which can now be considered as efficient alternatives to BP.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Although NREP subscribes to the predictive processing framework, it is not wedded to any specific process-theoretic description of the computational architecture underpinning such inferences.
“Although NREP subscribes to the predictive processing framework, it is not wedded to any specific process-theoretic description of the computational architecture underpinning such inferences (see, e.g., Brucklacher, Lee, et al., 2025; Salvatori et al., 2021; Spratling, 2017; Sprevak & Smith, 2023).”
From Integrated information and predictive processing theories of consciousness: An adversarial collaborative review (2026) · cites this paper as Salvatori et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 32 · Cited by 16
Research works printed but not linked: 18
- [1][1] J. Aljadeff, J. D'amour, R. E. Field, R. C. Froemke, and C. Clopath. Cortical credit assignment by Hebbian, neuromodulatory and inhibitory plasticity. arXiv:1911.00307 , 2019. - · arXiv
- [4][4] R. Bogacz. A tutorial on the free-energy framework for modelling perception and learning. Journal of Mathematical Psychology , 76:198-211, 2017. -
- [6][6] K. Friston. Learning and inference in the brain. Neural Networks , 16(9):1325-1352, 2003. -
- [7][7] K. Friston. A theory of cortical responses. Philos. Trans. R. Soc. Lond., B, Biol. Sci. , 360(1456):815-836, 2005. -
- [8][8] J. Hohwy, A. Roepstorff, and K. Friston. Predictive coding explains binocular rivalry: An epistemological review. Cognition , 108, 2008. -
- [10][10] R. Kanai, Y. Komura, S. Shipp, and K. Friston. Cerebral hierarchies: Predictive processing, precision and the pulvinar. Philos. Trans. R. Soc. Lond., B, Biol. Sci. , 370(1668), 2015. -
- [13][13] D.-H. Lee, S. Zhang, A. Fischer, and Y. Bengio. Difference target propagation. In Proc. ECMLPKDD , pages 498-515, 2015. -
- [15][15] W. Lotter, G. Kreiman, and D. Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv:1605.08104 , 2016. - · arXiv
- [17][17] A. Nøkland and L. H. Eidnes. Training neural networks with local error signals. arXiv:1901.06656 , 2019. - · arXiv
- [18][18] A. G. Ororbia, P. Haffner, D. Reitter, and C. L. Giles. Learning to adapt by minimizing discrepancy. arXiv:1711.11542 , 2017. - · arXiv
- [19][19] A. G. Ororbia and A. Mali. Biologically motivated algorithms for propagating local target representations. In Proc. AAAI , 2019. -
- [21][21] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. Nature , 323(6088), 1986. -
- [23][23] A. Sedghi, V. Gupta, and P. Long. The singular values of convolutional layers. In Proc. ICLR , 2020. -
- [24][24] Y. Song, T. Lukasiewicz, Z. Xu, and R. Bogacz. Can the brain do backpropagation?-exact implementation of backpropagation in predictive coding networks. NeurIPS , 2020. -
- [25][25] X. Wang, X. Lin, and X. Dang. Supervised learning in spiking neural networks: A review of algorithms and evaluations. Neural Networks , 2020. -
- [28][28] P. Werbos. New tools for prediction and analysis in the behavioral sciences. Ph.D. Dissertation, Harvard University , 1974. -
- [29][29] J. C. Whittington and R. Bogacz. An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity. Neural Computation , 29(5):1229-1262, 2017. -
- [31][31] X. Xie and H. S. Seung. Equivalence of backpropagation and contrastive Hebbian learning in a layered network. Neural Computation , 15(2):441-454, 2003. -
- Authors
- Salvatori, T., Song, Y., Lukasiewicz, T., Bogacz, R., & Xu, Z.
- Affiliations
- University of Oxford (GB)
- Hebei University of Technology (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2103.03725
- OpenAlex
- W3133991449
- Cited by
- 16
- References
- 32
- Citations by year
- 2026: 2 · 2024: 3 · 2023: 4 · 2022: 4 · 2021: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Engineering and Physical Sciences Research Council (EP/N510129/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural dynamics and brain function
- Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
- Keywords
- predictive coding, backpropagation, convolutional neural networks, recurrent neural networks, biologically plausible learning, weight updating
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
