Continuous learning in single-incremental-task scenarios
Open accessAbstract
It was recently shown that architectural, regularization and rehearsal strategies can be used to train deep models sequentially on a number of disjoint tasks without forgetting previously acquired knowledge. However, these strategies are still unsatisfactory if the tasks are not disjoint but constitute a single incremental task (e.g., class-incremental learning). In this paper we point out the differences between multi-task and single-incremental-task scenarios and show that well-known approaches such as LWF, EWC and SI are not ideal for incremental task scenarios. A new approach, denoted as AR1, combining architectural and regularization strategies is then specifically proposed. AR1 overhead (in terms of memory and computation) is very small thus making it suitable for online learning. When tested on CORe50 and iCIFAR-100, AR1 outperformed existing regularization strategies by a good margin.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
To avert catastrophic forgetting, various algorithms employ task-specific components or design separate networks for each task.
“To avert catastrophic forgetting, various algorithms[11, 48, 49] employ task-specific components or design separate networks for each task.”
From Few-Shot Class-Incremental Learning for Classification and Object Detection: A Survey (2025) · cites this paper as [49]
Section 1.1 describes architectural strategies that mitigate forgetting and gives Progressive Neural Networks as an example using parameter freezing and network expansion. Section 1.2 also describes a separate classification head for each task in multi-task learning. These passages support task-specific components, but they do not establish that algorithms design an entirely separate network for each task. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Part of this paper’s reference list could not be split into single entries.
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899 · see citing works
iCaRL: Incremental Classifier and Representation Learning
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., & Lampert, C. H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.587 · Cited by 3,988 · see citing works
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770 · see citing works
Memory Aware Synapses: Learning What (not) to Forget
Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., & Tuytelaars, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_9 · Cited by 1,590 · see citing works
Measuring Catastrophic Forgetting in Neural Networks
Kemker, R., McClure, M., Abitino, A., Hayes, T., & Kanan, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11651 · Cited by 672 · see citing works
Growing a Brain: Fine-Tuning by Increasing Model Capacity
Wang, Y.-X., Ramanan, D., & Hebert, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.323 · Cited by 154 · cited 1× · see citing works
“rate and the representation cannot be further adapted to new data. To deal with this problem we believe that the incremental lateral expansion of some intermediate layers, similar to that proposed in [32] can be a valid approach. Class-incremental learning (NC update content type) is only one of the cases of interest…”
Background
Flagged first, then by citations
Research works printed but not linked: 20
- [2][2] Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128–135, 1999. 20
- [3][3] Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks. arXiv preprint arXiv:1312.6211, 2013. · arXiv
- [5][5] Tyler L. Hayes, Nathan D. Cahill, and Christopher Kanan. Memory Efficient Experience Replay for Streaming Learning. arXiv preprint arXiv:1809.05922, 2018. · arXiv
- [8][8] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network. In NIPS Deep Learning and Representation Learning Workshop (2015), pages 1–9, 2015.
- [9][9] Ferenc Huszár. Note on the quadratic penalties in elastic weight consolidation. Proceedings of the National Academy of Sciences of the United States of America, 115(11):E2496–E2497, mar 2018.
- [10][10] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional Architecture for Fast Feature Embedding. Proceedings of the ACM International Conference on Multimedia, pages 675–678, 2014.
- [11][11] Ronald Kemker and Christopher Kanan. FearNet: Brain-Inspired Model For Incremental Learning. In International Conference on Learning Representations (ICLR2018), pages 1–16, Vancouver, Canada, 2018.
- [15][15] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Reply to Huszár: The elastic weight consolidation penalty is empirically valid. Proceedings of the National Academy of Sciences of the United States of America, 115(11):E2498, mar 2018.
- [16][16] Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. PhD thesis, 2009.
- [17][17] Zhizhong Li and Derek Hoiem. Learning without forgetting. In 14th European Conference on Computer Vision (ECCV 2016), volume 9908 LNCS, pages 614–629, Amsterdam, Netherlands, 2016.
- [18][18] Min Lin, Qiang Chen, and Shuicheng Yan. Network In Network. In International Conference on Learning Representations (ICLR 2014), 2014.
- [19][19] Vincenzo Lomonaco and Davide Maltoni. CORe50: a New Dataset and Benchmark for Continuous Object Recognition. In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg, editors, Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 17–26. PMLR, 2017.
- [20][20] Vincenzo Lomonaco and Davide Maltoni. CORe50: a New Dataset and Benchmark for Continuous Object Recognition. arXiv preprint arXiv:1705.03550, 2017. · arXiv
- [21][21] David Lopez-paz and Marc'Aurelio Ranzato. Gradient Episodic Memory for Continuum Learning. In Advances in neural information processing systems (NIPS 2017), 2017. 21
- [22][22] Davide Maltoni and Vincenzo Lomonaco. Semi-supervised Tuning from Temporal Coherence. In 23rd International Conference on Pattern Recognition (ICPR 2016), pages 2509–2514, 2016.
- [25][25] German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Con- tinual Lifelong Learning with Neural Networks: A Review. arXiv preprint arXiv:1802.07569, 2018. · arXiv
- [26][26] German I. Parisi, Jun Tani, Cornelius Weber, and Stefan Wermter. Lifelong Learning of Spatiotemporal Representations with Dual-Memory Recurrent Self-Organization. arXiv preprint arXiv:1805.10966, pages 1–20, 2018. · arXiv
- [27][27] R Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285–308, apr 1990.
- [29][29] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive Neural Networks. arXiv preprint arXiv:1606.04671, 2016. · arXiv
- [31][31] C Szegedy, W Liu, Y Jia, and P Sermanet. Going deeper with convolutions. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Cited by
Sorted by citations
Continual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.01.012 · Cited by 3,117 · see citing works
Three types of incremental learning
van de Ven, G. M., Tuytelaars, T., & Tolias, A. S. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00568-3 · Cited by 665 · see citing works
Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
Hsu, Y.-C., Liu, Y.-C., Ramasamy, A., & Kira, Z. (2018) · arXiv · doi:10.48550/arxiv.1810.12488 · Cited by 211 · see citing works
Topology-Preserving Class-Incremental Learning
Tao, X., Chang, X., Hong, X., Wei, X., & Gong, Y. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58529-7_16 · Cited by 162 · see citing works
An Appraisal of Incremental Learning Methods
Luo, Y., Yin, L., Bai, W., & Mao, K. (2020) · Entropy · doi:10.3390/e22111190 · Cited by 120 · see citing works
Few-Shot Class-Incremental Learning for Classification and Object Detection: A Survey
Zhang, J., Liu, L., Silvén, O., Pietikäinen, M., & Hu, D. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3529038 · Cited by 84 · see citing works
“To avert catastrophic forgetting, various algorithms [11], [48], [49] employ task-specific components or design separate networks for each task.”
Robustifying the Deployment of tinyML Models for Autonomous mini-vehicles
de Prado, M., Rusci, M., Capotondi, A., Donze, R., Benini, L., & Pazos, N. (2021) · Sensors · doi:10.3390/s21041339 · Cited by 57 · see citing works
- Authors
- Maltoni, D., & Lomonaco, V.
- Affiliations
- University of Bologna (IT)
- Corresponding author
- Maltoni, Davide
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2019-08
- Volume
- 116
- Pages
- 56-73
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W2809425026
- Cited by
- 309
- References
- 8
- Citations by year
- 2026: 12 · 2025: 33 · 2024: 46 · 2023: 43 · 2022: 54 · 2021: 66 · 2020: 35 · 2019: 16 · 2018: 4
- FWCI
- 19.50
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Neural Networks (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- continuous learning, class-incremental learning, catastrophic forgetting, architectural strategies, rehearsal strategies, online learning, multi-task learning
- MeSH terms
- Deep Learning, Humans, Photic Stimulation, Recognition, Psychology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
