Self-Net: Lifelong Learning via Continual Self-Modeling
Open accessAbstract
Learning a set of tasks over time, also known as continual learning (CL), is one of the most challenging problems in artificial intelligence. While recent approaches achieve some degree of CL in deep neural networks, they either (1) store a new network (or an equivalent number of parameters) for each new task, (2) store training data from previous tasks, or (3) restrict the network's ability to learn new tasks. To address these issues, we propose a novel framework, Self-Net, that uses an autoencoder to learn a set of low-dimensional representations of the weights learned for different tasks. We demonstrate that these low-dimensional vectors can then be used to generate high-fidelity recollections of the original weights. Self-Net can incorporate new tasks over time with little retraining, minimal loss in performance for older tasks, and without storing prior training data. We show that our technique achieves over 10X storage compression in a continual fashion, and that it outperforms state-of-the-art approaches on numerous datasets, including continual versions of MNIST, CIFAR10, CIFAR100, Atari, and task-incremental CORe50. To the best of our knowledge, we are the first to use autoencoders to sequentially encode sets of network weights to enable continual learning.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Self-Net's parameters grow only logarithmically with the number of tasks, and it has achieved good results in storage compression.
“Since parameters grow only logarithmically with the number of tasks, Self-Net has achieved good results in storage compression.”
From An Appraisal of Incremental Learning Methods (2020) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471 · see citing works
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899 · see citing works
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · see citing works
Li, Z., & Hoiem, D. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2773081 · Cited by 4,202 · see citing works
iCaRL: Incremental Classifier and Representation Learning
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., & Lampert, C. H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.587 · Cited by 3,988 · see citing works
Continual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.01.012 · Cited by 3,117 · see citing works
Continual Learning Through Synaptic Intelligence
Zenke, F., Poole, B., & Ganguli, S. (2017) · arXiv · doi:10.48550/arxiv.1703.04200 · Cited by 1,784 · see citing works
Tutorial on Variational Autoencoders
Doersch, C. (2016) · arXiv · doi:10.48550/arxiv.1606.05908 · Cited by 1,470 · see citing works
Gradient Episodic Memory for Continual Learning
Lopez-Paz, D., & Ranzato, M. (2017) · arXiv · doi:10.48550/arxiv.1706.08840 · Cited by 1,403 · see citing works
Interplay of Hippocampus and Prefrontal Cortex in Memory
Preston, A. R., & Eichenbaum, H. (2013) · Current Biology · doi:10.1016/j.cub.2013.05.041 · Cited by 1,389 · see citing works
Continual Learning with Deep Generative Replay
Shin, H., Lee, J. K., Kim, J., & Kim, J. (2017) · arXiv · doi:10.48550/arxiv.1705.08690 · Cited by 1,385 · see citing works
Weight Uncertainty in Neural Networks
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1505.05424 · Cited by 1,342 · see citing works
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1606.04671 · Cited by 1,307 · see citing works
Contractive Auto-Encoders: Explicit Invariance During Feature Extraction
Rifai, S., Vincent, P., Muller, X., Glorot, X., & Bengio, Y. (2011) · Cited by 1,189 · see citing works
An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6211 · Cited by 1,089 · see citing works
Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval
Carr, M. F., Jadhav, S. P., & Frank, L. M. (2011) · Nature Neuroscience · doi:10.1038/nn.2732 · Cited by 958 · see citing works
The hippocampal memory indexing theory.
Teyler, T. J., & DiScenna, P. (1986) · Behavioral Neuroscience · doi:10.1037/0735-7044.100.2.147 · Cited by 871 · see citing works
Catastrophic forgetting, rehearsal and pseudorehearsal
ROBINS, A. (1995) · Connection Science · doi:10.1080/09540099550039318 · Cited by 833 · see citing works
Lifelong Learning with Dynamically Expandable NetworksPreprint
Yoon, J., Yang, E., Lee, J., & Hwang, S. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.01547 · Cited by 786 · see citing works
What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated
Kumaran, D., Hassabis, D., & McClelland, J. L. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.05.004 · Cited by 736 · see citing works
Measuring Catastrophic Forgetting in Neural Networks
Kemker, R., McClure, M., Abitino, A., Hayes, T., & Kanan, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11651 · Cited by 672 · see citing works
Li, Z., & Hoiem, D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_37 · Cited by 659 · see citing works
Expert Gate: Lifelong Learning with a Network of Experts
Aljundi, R., Chakravarty, P., & Tuytelaars, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.753 · Cited by 619 · see citing works
Variational Continual Learning
Turner, R., Bui, T., Li, Y., & Cuong, N. (2019) · Apollo - University of Cambridge Repository · doi:10.17863/cam.35471 · Cited by 438 · see citing works
Progress & Compress: A scalable framework for continual learningPreprint
Schwarz, J., Luketina, J., Czarnecki, W. M., Grabska‐Barwińska, A., Teh, Y. W., Pascanu, R., & Hadsell, R. T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.06370 · Cited by 422 · see citing works
FearNet: Brain-Inspired Model for Incremental LearningPreprint
Kemker, R., & Kanan, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.10563 · Cited by 376 · see citing works
Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization
Masse, N. Y., Grant, G. D., & Freedman, D. J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1803839115 · Cited by 260 · see citing works
Online Meta-LearningPreprint
Finn, C., Rajeswaran, A., Kakade, S. M., & Levine, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.08438 · Cited by 221 · see citing works
Generative replay with feedback connections as a general strategy for continual learningPreprint
van de Ven, G. M., & Tolias, A. S. (2018) · Lirias · doi:10.48550/arxiv.1809.10635 · Cited by 194 · see citing works
Continual learning with hypernetworks
von Oswald, J., Henning, C., Grewe, B. F., & Sacramento, J. (2019) · Zurich Open Repository and Archive (University of Zurich) · doi:10.3929/ethz-b-000465911 · Cited by 189 · see citing works
Measuring Catastrophic Forgetting in Neural NetworksPreprint
Kemker, R., McClure, M., Abitino, A., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.02072 · Cited by 180 · see citing works
Online Incremental Feature Learning with Denoising Autoencoders
Zhou, G., Sohn, K., & Lee, H. (2012) · Cited by 162 · see citing works
CORe50: a New Dataset and Benchmark for Continuous Object Recognition
Lomonaco, V., & Maltoni, D. (2017) · arXiv · doi:10.48550/arxiv.1705.03550 · Cited by 135 · see citing works
Note on the quadratic penalties in elastic weight consolidation
Huszár, F. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1717042115 · Cited by 134 · see citing works
Lifelong Learning of Spatiotemporal Representations With Dual-Memory Recurrent Self-Organization
Parisi, G. I., Tani, J., Weber, C., & Wermter, S. (2018) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2018.00078 · Cited by 126 · see citing works
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLPreprint
Nagabandi, A., Finn, C., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.07671 · Cited by 120 · see citing works
Lifelong Learning of Spatiotemporal Representations With Dual-Memory Recurrent Self-Organization
Parisi, G. I., Tani, J., Weber, C., & Wermter, S. (2018) · Institutional Repositories DataBase (IRDB) · Cited by 120 · see citing works
Ramapuram, J., Gregorová, M., & Kalousis, A. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.02.115 · Cited by 112 · see citing works
Task Agnostic Continual Learning via Meta LearningPreprint
He, X., Sygnowski, J., Galashov, A., Rusu, A. A., Teh, Y. W., & Pascanu, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05201 · Cited by 59 · see citing works
Improved multitask learning through synaptic intelligence.Preprint
Zenke, F., Poole, B., & Ganguli, S. (2017) · arXiv (Cornell University) · Cited by 29 · see citing works
Scalable Recollections for Continual Lifelong Learning
Riemer, M., Klinger, T., Bouneffouf, D., & Franceschini, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33011352 · Cited by 11 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Continual learning with hypernetworks
von Oswald, J., Henning, C., Grewe, B. F., & Sacramento, J. (2019) · Zurich Open Repository and Archive (University of Zurich) · doi:10.3929/ethz-b-000465911 · Cited by 189 · see citing works
An Appraisal of Incremental Learning Methods
Luo, Y., Yin, L., Bai, W., & Mao, K. (2020) · Entropy · doi:10.3390/e22111190 · Cited by 120 · see citing works
Montero, A. M. B., Bibal, A., Huet‐Dastarac, M., Draguet, C., Valdés, G., Nguyen, D. T. D., Willems, S., Vandewinckele, L., Holmström, M., Löfman, F., et al. (2022) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ac678a · Cited by 87 · see citing works
Continual learning in sensor-based human activity recognition: An empirical benchmark analysis
Jha, S., Schiemer, M., Zambonelli, F., & Ye, J. (2021) · Information Sciences · doi:10.1016/j.ins.2021.04.062 · Cited by 63 · see citing works
Advancements in AI-driven diagnostic radiology: Enhancing accuracy and efficiency
Alanazi, M. M. F., Almutairi, S., Alarjani, N., Alghaylan, . Y. A., Aljawhari, . S. M., & Alkhulaifi, . A. S. (2024) · International Journal of Health Sciences · doi:10.53730/ijhs.v8ns1.14928 · Cited by 20 · see citing works
Mousavi, M., & Estrada, R. G. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00108 · Cited by 10 · see citing works
Self-citationContinual Learning with Deep Artificial NeuronsPreprint
Camp, B., Mandivarapu, J. K., & Estrada, R. G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.07035 · Cited by 8 · see citing works
Self-citationDetecting and Learning the Unknown in Semantic Segmentation
Chan, R., Uhlemeyer, S., Rottmann, M., & Gottschalk, H. (2022) · Springer eBooks · doi:10.1007/978-3-031-01233-4_10 · Cited by 4 · see citing works
Grid Synchronization using Machine Learning
Bhandare, R., Vaidya, K. P., & Suraparaju, K. R. (2022) · IEEE Mysore Sub Section International Conference (MysuruCon) · doi:10.1109/mysurucon55714.2022.9972611 · Cited by 3 · see citing works
Savage, N. J. (2022) · Nature · doi:10.1038/d41586-022-01962-y · Cited by 2 · see citing works
Experimental study of rehearsal-based incremental classification of document streams
Malik, U., Visani, M., Sidère, N., Coustaty, M., & Joseph, A. (2024) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-024-00467-w · Cited by 1 · see citing works
Remembering Transformer for Continual Learning
Sun, Y., Fujisawa, I., Juliani, A., Sakuma, J., & Kanai, R. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227313 · Cited by 1 · see citing works
Incremental Learning With Adaptive Model Search and a Nominal Loss Model
Ahn, C., Kim, E., & Oh, S. (2022) · IEEE Access · doi:10.1109/access.2022.3149598 · Cited by 1 · see citing works
Memory Consolidation with Orthogonal Gradients for avoiding Catastrophic ForgettingPreprint
Kanagamani, T., Krishnamurthy, R., Chakravarthy, V. S., Ravindran, B., & Menon, R. N. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.02.25.481890
Improve the Performance and Stability of Incremental Learning by a Similarity Harmonizing Mechanism
Ma, J., Liao, M., & Zhang, L. (2022) · IEEE Access · doi:10.1109/access.2022.3219502
- Authors
- Mandivarapu, J. K., Camp, B., & Estrada, R.
- Affiliations
- Georgia State University (US)
- Corresponding author
- Estrada, Rolando
- Venue
- Frontiers in Artificial Intelligence
- Publisher
- Frontiers Media SA
- Published
- 2020-04-09
- Volume
- 3
- Pages
- 19-19
- ISSN
- 2624-8212 (ISSN-L)
- PMC
- PMC7861283
- OpenAlex
- W3015684125
- Cited by
- 14
- References
- 44
- Citations by year
- 2025: 1 · 2024: 2 · 2022: 7 · 2021: 2 · 2020: 2
- FWCI
- 1.08
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Artificial Intelligence (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Directorate for Computer and Information Science and Engineering (1849946)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- continual learning, lifelong learning, self-modeling, autoencoder, storage compression, catastrophic forgetting, task-incremental learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
