Continual Learning Through Synaptic Intelligence
Open accessAbstract
While deep learning has led to remarkable advances across diverse applications, it struggles in domains where the data distribution changes over the course of learning. In stark contrast, biological neural networks continually adapt to changing domains, possibly by leveraging complex molecular machinery to solve many tasks simultaneously. In this study, we introduce intelligent synapses that bring some of this biological complexity into artificial neural networks. Each synapse accumulates task relevant information over time, and exploits this information to rapidly store new memories without forgetting old ones. We evaluate our approach on continual learning of classification tasks, and show that it dramatically reduces forgetting while maintaining computational efficiency.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019 · see citing works
Flagged first, then by citations
Research works printed but not linked: 25
- Benna, Marcus K. and Fusi, Stefano. Computational principles of synaptic memory consolidation. Nat Neurosci , advance online publication, October 2016. ISSN 10976256. doi: 10.1038/nn.4401. · DOI
- Choy, Min Chee, Srinivasan, Dipti, and Cheu, Ruey Long. Neural networks for continuous online learning and control. IEEE Trans Neural Netw , 17(6):1511-1531, November 2006. ISSN 1045-9227. doi: 10.1109/TNN. 2006.881710. · DOI
- Fukushima, Kunihiko and Miyake, Sei. Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Visual Pattern Recognition. In Competition and Cooperation in Neural Nets , pp. 267-285. Springer, Berlin, Heidelberg, 1982. DOI: 10.1007/978-3-64246466-9 18. · DOI
- Fusi, Stefano, Drew, Patrick J., and Abbott, Larry F. Cascade models of synaptically stored memories. Neuron , 45(4):599-611, February 2005. ISSN 0896-6273. doi: 10.1016/j.neuron.2005.02.001. · DOI · Paper page
- Goodfellow, Ian J., Mirza, Mehdi, Xiao, Da, Courville, Aaron, and Bengio, Yoshua. An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks. arXiv:1312.6211 [cs, stat] , December 2013. arXiv: 1312.6211. · arXiv
- Hinton, Geoffrey, Vinyals, Oriol, and Dean, Jeff. Distilling the knowledge in a neural network. NIPS Deep Learning and Representation Learning Workshop , 2014.
- Jung, Heechul, Ju, Jeongwoo, Jung, Minju, and Kim, Junmo. Less-forgetting Learning in Deep Neural Networks. arXiv:1607.00122 [cs] , July 2016. arXiv: 1607.00122. · arXiv
- Kingma, Diederik and Ba, Jimmy. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs] , December 2014. arXiv: 1412.6980. · arXiv
- Kirkpatrick, James, Pascanu, Razvan, Rabinowitz, Neil, Veness, Joel, Desjardins, Guillaume, Rusu, Andrei A., Milan, Kieran, Quan, John, Ramalho, Tiago, GrabskaBarwinska, Agnieszka, Hassabis, Demis, Clopath, Claudia, Kumaran, Dharshan, and Hadsell, Raia. Overcoming catastrophic forgetting in neural networks. PNAS , pp. 201611835, March 2017. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.1611835114. · DOI · Paper page
- Lahiri, Subhaneil and Ganguli, Surya. A memory frontier for complex synapses. In Advances in Neural Information Processing Systems , volume 26, pp. 1034-1042, Tahoe, USA, 2013. Curran Associates, Inc.
- LeCun, Yann, Bengio, Yoshua, and Hinton, Geoffrey. Deep learning. Nature , 521(7553):436-444, May 2015. ISSN 0028-0836. doi: 10.1038/nature14539. · DOI · Paper page
- Li, Zhizhong and Hoiem, Derek. Learning without forgetting. In European Conference on Computer Vision , pp. 614-629. Springer, 2016.
- Martens, James. Second-order optimization for neural networks . PhD thesis, University of Toronto, 2016.
- Martens, James, Sutskever, Ilya, and Swersky, Kevin. Estimating the hessian by back-propagating curvature. arXiv preprint arXiv:1206.6464 , 2012. · arXiv
- Montgomery, Johanna M. and Madison, Daniel V. StateDependent Heterogeneity in Synaptic Depression between Pyramidal Cell Pairs. Neuron , 33(5):765-777, February 2002. ISSN 0896-6273. doi: 10.1016/ S0896-6273(02)00606-2.
- Pascanu, Razvan and Bengio, Yoshua. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584 , 2013. · arXiv
- Razavian, Ali Sharif, Azizpour, Hossein, Sullivan, Josephine, and Carlsson, Stefan. Cnn features off-theshelf: an astounding baseline for recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pp. 806-813, 2014.
- Redondo, Roger L. and Morris, Richard G. M. Making memories last: the synaptic tagging and capture hypothesis. Nat Rev Neurosci , 12(1):17-30, January 2011. ISSN 1471-003X. doi: 10.1038/nrn2963. · DOI · Paper page
- Rosenblatt, Frank. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological review , 65(6):386, 1958.
- Rusu, Andrei A., Rabinowitz, Neil C., Desjardins, Guillaume, Soyer, Hubert, Kirkpatrick, James, Kavukcuoglu, Koray, Pascanu, Razvan, and Hadsell, Raia. Progressive Neural Networks. arXiv:1606.04671 [cs] , June 2016. arXiv: 1606.04671. · arXiv
- Srivastava, Rupesh K, Masci, Jonathan, Kazerounian, Sohrob, Gomez, Faustino, and Schmidhuber, Juergen. Compete to Compute. In Burges, C. J. C., Bottou, L., Welling, M., Ghahramani, Z., and Weinberger, K. Q. (eds.), Advances in Neural Information Processing Systems 26 , pp. 2310-2318. Curran Associates, Inc., 2013.
- Yosinski, Jason, Clune, Jeff, Bengio, Yoshua, and Lipson, Hod. How transferable are features in deep neural networks? In Advances in neural information processing systems , pp. 3320-3328, 2014.
- Zenke, Friedemann, Agnes, Everton J., and Gerstner, Wulfram. Diverse synaptic plasticity mechanisms orchestrated to form and retrieve memories in spiking neural networks. Nat Commun , 6, April 2015. doi: doi: 10.1038/ncomms7922. · DOI
- Zhou, Qiang, Tao, Huizhong W., and Poo, Mu-Ming. Reversal and Stabilization of Synaptic Modifications in a Developing Visual System. Science , 300(5627):19531957, June 2003. doi: 10.1126/science.1082212. · DOI
- Ziegler, Lorric, Zenke, Friedemann, Kastner, David B., and Gerstner, Wulfram. Synaptic Consolidation: From Synapses to Behavioral Modeling. J Neurosci , 35(3): 1319-1334, January 2015. ISSN 0270-6474, 1529-2401. doi: 10.1523/JNEUROSCI.3989-14.2015. · DOI
Cited by
Sorted by citations
Continual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.01.012 · Cited by 3,117 · see citing works
Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
Chaudhry, A., Dokania, P. K., Ajanthan, T., & Torr, P. H. S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_33 · Cited by 1,091 · see citing works
A Comprehensive Survey of Continual Learning: Theory, Method and Application
Wang, L., Zhang, X., Su, H., & Zhu, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3367329 · Cited by 1,113 · see citing works
Three types of incremental learning
van de Ven, G. M., Tuytelaars, T., & Tolias, A. S. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00568-3 · Cited by 665 · see citing works
Task representations in neural networks trained to perform many cognitive tasks
Yang, G. R., Joglekar, M. R., Song, H. F., Newsome, W. T., & Wang, X.-J. (2019) · Nature Neuroscience · doi:10.1038/s41593-018-0310-2 · Cited by 549 · see citing works
Brain-inspired replay for continual learning with artificial neural networks
van de Ven, G. M., Siegelmann, H. T., & Tolias, A. S. (2020) · Nature Communications · doi:10.1038/s41467-020-17866-2 · Cited by 474 · see citing works
If deep learning is the answer, what is the question?
Saxe, A., Nelli, S., & Summerfield, C. (2021) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-00395-8 · Cited by 397 · see citing works
Prototype Augmentation and Self-Supervision for Incremental Learning
Zhu, F., Zhang, X.-Y., Wang, C., Yin, F., & Liu, C.-L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00581 · Cited by 427 · see citing works
Self-Supervised GANs via Auxiliary Rotation Loss
Chen, T., Zhai, X., Ritter, M., Lucic, M., & Houlsby, N. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01243 · Cited by 328 · see citing works
The role of population structure in computations through neural dynamics
Dubreuil, A., Valente, A., Beiran, M., Mastrogiuseppe, F., & Ostojic, S. (2022) · Nature Neuroscience · doi:10.1038/s41593-022-01088-4 · Cited by 237 · see citing works
Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
Hsu, Y.-C., Liu, Y.-C., Ramasamy, A., & Kira, Z. (2018) · arXiv · doi:10.48550/arxiv.1810.12488 · Cited by 211 · see citing works
Lifelong Learning via Progressive Distillation and Retrospection
Hou, S., Pan, X., Loy, C. C., Wang, Z., & Lin, D. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_27 · Cited by 190 · see citing works
Deep Model Poisoning Attack on Federated Learning
Zhou, X., Xu, M., Wu, Y., & Zheng, N. (2021) · Future Internet · doi:10.3390/fi13030073 · Cited by 170 · see citing works
Topology-Preserving Class-Incremental Learning
Tao, X., Chang, X., Hong, X., Wei, X., & Gong, Y. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58529-7_16 · Cited by 162 · see citing works
Loss of plasticity in deep continual learning
Dohare, S., Hernandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., & Sutton, R. S. (2024) · Nature · doi:10.1038/s41586-024-07711-7 · Cited by 172 · see citing works
An Appraisal of Incremental Learning Methods
Luo, Y., Yin, L., Bai, W., & Mao, K. (2020) · Entropy · doi:10.3390/e22111190 · Cited by 120 · see citing works
Dynamic Memory to Alleviate Catastrophic Forgetting in Continual Learning with Medical Imaging
Perkonigg, M., Hofmanninger, J., Herold, C. J., Brink, J. A., Pianykh, O., Prosch, H., & Langs, G. (2021) · Nature Communications · doi:10.1038/s41467-021-25858-z · Cited by 106 · see citing works
Replay in Deep Learning: Current Approaches and Missing Biological Elements
Hayes, T. L., Krishnan, G. P., Bazhenov, M., Siegelmann, H. T., Sejnowski, T. J., & Kanan, C. (2021) · Neural Computation · doi:10.1162/neco_a_01433 · Cited by 106 · see citing works
Continual Relation Learning via Episodic Memory Activation and Reconsolidation
Han, X., Dai, Y., Gao, T., Lin, Y., Liu, Z., Li, P., Sun, M., & Zhou, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.573 · Cited by 104 · see citing works
Inferring neural activity before plasticity as a foundation for learning beyond backpropagation
Song, Y., Millidge, B., Salvatori, T., Lukasiewicz, T., Xu, Z., & Bogacz, R. (2024) · Nature Neuroscience · doi:10.1038/s41593-023-01514-1 · Cited by 85 · see citing works
Clune, J. (2019) · arXiv · doi:10.48550/arxiv.1905.10985 · Cited by 82 · see citing works
Learning with Selective Forgetting
Shibata, T., Irie, G., Ikami, D., & Mitsuzumi, Y. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/137 · Cited by 47 · see citing works
Continual Learning for Generative Retrieval over Dynamic Corpora
Chen, J., Zhang, R., Guo, J., de Rijke, M., Chen, W., Fan, Y., & Cheng, X. (2023) · Proceedings of the 32nd ACM International Conference on Information and Knowledge Management · doi:10.1145/3583780.3614821 · Cited by 40 · see citing works
DFIL: Deepfake Incremental Learning by Exploiting Domain-invariant Forgery Clues
Pan, K., Yin, Y., Wei, Y., Lin, F., Ba, Z., Liu, Z., Wang, Z., Cavallaro, L., & Ren, K. (2023) · Proceedings of the 31st ACM International Conference on Multimedia · doi:10.1145/3581783.3612377 · Cited by 37 · see citing works
Continual learning of longitudinal health records
Armstrong, J., & Clifton, D. A. (2022) · 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI) · doi:10.1109/bhi56158.2022.9926878 · Cited by 21 · see citing works
Open-world continual learning: Unifying novelty detection and continual learning
Kim, G., Xiao, C., Konishi, T., Ke, Z., & Liu, B. (2025) · Artificial Intelligence · doi:10.1016/j.artint.2024.104237 · Cited by 19 · see citing works
Data Augmented Flatness-aware Gradient Projection for Continual Learning
Yang, E., Shen, L., Wang, Z., Liu, S., Guo, G., & Wang, X. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00518 · Cited by 18 · see citing works
Self-Net: Lifelong Learning via Continual Self-Modeling
Mandivarapu, J. K., Camp, B., & Estrada, R. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.00019 · Cited by 14 · see citing works
Input-driven dynamics for robust memory retrieval in Hopfield networks
Betteti, S., Baggio, G., Bullo, F., & Zampieri, S. (2025) · Science Advances · doi:10.1126/sciadv.adu6991 · Cited by 12 · see citing works
Revisiting Flatness-Aware Optimization in Continual Learning With Orthogonal Gradient Projection
Yang, E., Shen, L., Wang, Z., Liu, S., Guo, G., Wang, X., & Tao, D. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3539019 · Cited by 5 · see citing works
The Future of Continual Learning in the Era of Foundation Models: Three Key Directions
Bell, J., Quarantiello, L., Coleman, E. N., Li, L., Li, M., Madeddu, M., Piccoli, E., & Lomonaco, V. (2025) · arXiv · doi:10.48550/arxiv.2506.03320 · Cited by 2
Zhou, S., & Li, Q. (2025) · Scientific Reports · doi:10.1038/s41598-025-31685-9
UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models
Gu, X., Huang, Z., Gu, J.-C., & Zhang, K. (2025) · arXiv · doi:10.48550/arxiv.2505.14679
Agent-Dice: Disentangling Knowledge Updates via Geometric Consensus for Agent Continual Learning
Wu, Z., Lou, X., Ma, X., Li, Y., Liu, W., Zhang, W., Wang, J., & Zhang, Z. (2026) · arXiv · doi:10.48550/arxiv.2601.03641
- Authors
- Zenke, F., Poole, B., & Ganguli, S.
- Affiliations
- Stanford University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- Volume
- 70
- Pages
- 3987-3995
- arXiv
- 1703.04200
- PMC
- PMC6944509
- OpenAlex
- W2737492962
- Cited by
- 1,784
- References
- 1
- Citations by year
- 2026: 3 · 2025: 173 · 2024: 285 · 2023: 335 · 2022: 259 · 2021: 333 · 2020: 220 · 2019: 129 · 2018: 42 · 2017: 3 · 2012: 1
- FWCI
- 64.27
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- PubMed
- Funders
- National Science Foundation
- McKnight Foundation
- Wellcome Trust
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
- Office of Naval Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- continual learning, synaptic plasticity, forgetting, memory consolidation, classification tasks, computational efficiency, artificial neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
