A predictive-coding network that is both discriminative and generative
Abstract
Predictive coding (PC) networks are a biologically interesting class of neural networks. Their layered hierarchy mimics the reciprocal connectivity pattern observed in the mammalian cortex, and they can be trained using local learning rules that approximate backpropagation (Bogacz, 2017 ). However, despite having feedback connections that enable information to flow down the network hierarchy, discriminative PC networks are not typically generative. Clamping the output class and running the network to equilibrium yields an input sample that usually does not resemble the training input. This letter studies this phenomenon and proposes a simple solution that promotes the generation of input samples that resemble the training inputs. Simple decay, a technique already in wide use in neural networks, pushes the PC network toward a unique minimum two-norm solution, and that unique solution provably (for linear networks) matches the training inputs. The method also vastly improves the samples generated for nonlinear networks, as we demonstrate on MNIST.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Predictive coding networks have recently been rediscovered as candidates for solving machine learning tasks and as a potential alternative to backpropagation.
“Furthermore, predictive coding networks have recently been rediscovered as potent candidates for solving machine learning tasks (e.g., the classification or generation of static images, Sun & Orchard, 2020; Song et al., 2020) and even a potential alternative to the backpropagation algorithm (Song et al., 2024; Millidge et al., 2022b; Whittington & Bogacz, 2017).”
From The generalized Hierarchical Gaussian Filter (2026) · cites this paper as Sun & Orchard, 2020
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · see citing works
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
Rao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483 · cited 3× · see citing works
“Moreover, the notion of higher-level generative feedback is consonant with receptive-field tuning, such as that observed in the end-stopping phenomenon (Rao & Ballard, 1999).”
BackgroundMethodInfluentialCanonical Microcircuits for Predictive Coding
Bastos, A. M., Usrey, W. M., Adams, R. A., Mangun, G. R., Fries, P., & Friston, K. J. (2012) · Neuron · doi:10.1016/j.neuron.2012.10.038 · Cited by 2,421 · cited 2× · see citing works
“A number of lines of neuroscientific evidence (Bartels, 2014; Bastos et al., 2012; Shipp, 2016) support the hypothesis that the cortex implements a predictive coding strategy.”
BackgroundOn the computational architecture of the neocortex: II The role of cortico-cortical loops
Mumford, D. (1992) · Biological Cybernetics · doi:10.1007/bf00198477 · Cited by 1,074 · see citing works
Generalization of back-propagation to recurrent neural networks
Pineda, F. J. (1987) · Physical Review Letters · doi:10.1103/physrevlett.59.2229 · Cited by 931 · see citing works
On the computational architecture of the neocortex: I. The role of the thalamo-cortical loop
Mumford, D. (1991) · Biological Cybernetics · doi:10.1007/bf00202389 · Cited by 929 · see citing works
Deep Predictive Coding Networks for Video Prediction and Unsupervised LearningPreprint
Lotter, W. E., Kreiman, G., & Cox, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08104 · Cited by 805 · see citing works
Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation
Scellier, B., & Bengio, Y. (2017) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2017.00024 · Cited by 504 · see citing works
Towards deep learning with segregated dendrites
Guerguiev, J., Lillicrap, T., & Richards, B. A. (2017) · eLife · doi:10.7554/elife.22901 · Cited by 373 · see citing works
O'Reilly, R. C. (1996) · Neural Computation · doi:10.1162/neco.1996.8.5.895 · Cited by 352 · see citing works
Neural elements for predictive coding
Shipp, S. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01792 · Cited by 340 · see citing works
To recognize shapes, first learn to generate images
Hinton, G. E. (2007) · Progress in brain research · doi:10.1016/s0079-6123(06)65034-6 · Cited by 304 · see citing works
Reading the mind's eye: Decoding category information during mental imagery
Reddy, L., Tsuchiya, N., & Serre, T. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.11.084 · Cited by 296 · see citing works
A tutorial on the free-energy framework for modelling perception and learning
Bogacz, R. (2017) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.11.003 · Cited by 283 · see citing works
Whittington, J. C. R., & Bogacz, R. (2017) · Neural Computation · doi:10.1162/neco_a_00949 · Cited by 278 · see citing works
Predictive coding as a model of biased competition in visual attention
Spratling, M. (2008) · Vision Research · doi:10.1016/j.visres.2008.03.009 · Cited by 240 · see citing works
Equivalence of Backpropagation and Contrastive Hebbian Learning in a Layered Network
Xie, X. S., & Seung, H. S. (2003) · Neural Computation · doi:10.1162/089976603762552988 · Cited by 190 · see citing works
Deep Predictive Coding NetworksPreprint
Chalasani, R., & Príncipe, J. C. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3541 · Cited by 52 · see citing works
Deep Predictive Coding Network with Local Recurrent Processing for Object RecognitionPreprint
Han, K., Wen, H., Zhang, Y., Fu, D., Culurciello, E., & Liu, Z. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.07526 · Cited by 46 · see citing works
Visual Perception: Early Visual Cortex Fills in the Gaps
Bartels, A. (2014) · Current Biology · doi:10.1016/j.cub.2014.05.055 · Cited by 10 · see citing works
A Deep Predictive Coding Network for Learning Latent RepresentationsPreprint
Dora, S., Pennartz, C. M. A., & Bohté, S. M. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/278218 · Cited by 5 · see citing works
Combating Adversarial Inputs Using a Predictive-Estimator Network
Orchard, J., & Castricato, L. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-70096-0_13 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
1 of the 25 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Hybrid predictive coding: Inferring, fast and slow
Tscshantz, A., Millidge, B., Seth, A. K., & Buckley, C. L. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011280 · Cited by 51 · see citing works
Predictive coding networks for temporal prediction
Millidge, B., Tang, M., Osanlouy, M., Harper, N. S., & Bogacz, R. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011183 · Cited by 44 · see citing works
Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?
Millidge, B., Salvatori, T., Song, Y., Bogacz, R., & Lukasiewicz, T. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/774 · Cited by 35 · see citing works
Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?
Millidge, B., Salvatori, T., Song, Y., Bogacz, R., & Lukasiewicz, T. (2022) · arXiv · doi:10.48550/arxiv.2202.09467 · Cited by 31 · see citing works
Learning probability distributions of sensory inputs with Monte Carlo predictive coding
Oliviers, G., Bogacz, R., & Meulemans, A. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012532 · Cited by 6 · see citing works
Deep predictive coding with bi-directional propagation for classification and reconstruction
Qiu, S., Bhattacharyya, S., Coyle, D., & Dora, S. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107785 · Cited by 5 · see citing works
Continual learning with a predictive coding based classifier
Qiu, S., Bhattacharyya, S., Coyle, D., & Dora, S. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.114265 · Cited by 2 · see citing works
Imagination, Agency, and Predictive Processing
Corlett, P. R. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780198872221.003.0004 · Cited by 2 · see citing works
The generalized Hierarchical Gaussian Filter
Weber, L. A., Waade, P. T., Legrand, N., Møller, A. H., Stephan, K. E., & Mathys, C. (2026) · doi:10.7554/elife.110174.1
The generalized Hierarchical Gaussian Filter
Weber, L. A., Waade, P. T., Legrand, N., Møller, A. H., Stephan, K. E., & Mathys, C. (2026) · eLife · doi:10.7554/elife.110174
Decay Makes Supervised Predictive Coding Generative
Sun, W. (2020) · UWSpace (University of Waterloo)
Self-citationDeveloping Variational Generative Models Using Predictive Coding
Qiu, S., Bhattacharyya, S., Coyle, D., & Dora, S. (2024) · Advances in intelligent systems and computing · doi:10.1007/978-3-031-78857-4_2
- Authors
- Sun, W., & Orchard, J.
- Affiliations
- University of Waterloo (CA)
- Corresponding author
- Sun, Wei
- Orchard, Jeff
- ORCID
- Orchard, Jeff
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2020-10
- Volume
- 32
- Issue
- 10
- Pages
- 1836-1862
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W3049639017
- Cited by
- 11
- References
- 25
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 4 · 2023: 1 · 2022: 1 · 2020: 1
- FWCI
- 0.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural dynamics and brain function
- Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
- Keywords
- predictive coding, feedback connections, local learning rules, sample generation, input reconstruction, linear networks, nonlinear networks, MNIST
- MeSH terms
- Discrimination Learning, Forecasting, Humans, Pattern Recognition, Automated, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
