A Strategy for an Uncompromising Incremental Learner
Open accessAbstract
Multi-class supervised learning systems require the knowledge of the entire range of labels they predict. Often when learnt incrementally, they suffer from catastrophic forgetting. To avoid this, generous leeways have to be made to the philosophy of incremental learning that either forces a part of the machine to not learn, or to retrain the machine again with a selection of the historic data. While these hacks work to various degrees, they do not adhere to the spirit of incremental learning. In this article, we redefine incremental learning with stringent conditions that do not allow for any undesirable relaxations and assumptions. We design a strategy involving generative models and the distillation of dark knowledge as a means of hallucinating data along with appropriate targets from past distributions. We call this technique, phantom sampling.We show that phantom sampling helps avoid catastrophic forgetting during incremental learning. Using an implementation based on deep neural networks, we demonstrate that phantom sampling dramatically avoids catastrophic forgetting. We apply these strategies to competitive multi-class incremental learning of deep neural networks. Using various benchmark datasets and through our strategy, we demonstrate that strict incremental learning could be achieved. We further put our strategy to test on challenging cases, including cross-domain increments and incrementing on a novel label space. We also propose a trivial extension to unbounded-continual learning and identify potential for future development.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In medical data processing, no memory of past classes is often due to privacy issues.
“For instance, in medical data processing [72], this is often due to privacy issues.”
From A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020) · cites this paper as [72]
In many applications, no memory of past classes is available.
“In many applications, no memory of past classes is available.”
From A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020) · cites this paper as [72]
Cites
(2014) · Cited by 33,419 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,268 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899 · see citing works
Reading digits in natural images with unsupervised feature learning
Netzer, Y. (2024) · doi:10.57702/e7wwo0zk · Cited by 4,261 · see citing works
FitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859 · see citing works
An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6211 · Cited by 1,089 · see citing works
Theano: A CPU and GPU Math Compiler in Python
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J. P., Warde-Farley, D., & Bengio, Y. (2010) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-92bf1922-003 · Cited by 1,063 · see citing works
An empirical evaluation of deep architectures on problems with many factors of variation
Larochelle, H., Erhan, D., Courville, A. C., Bergstra, J., & Bengio, Y. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273556 · Cited by 1,024 · see citing works
Learn++: an incremental learning algorithm for supervised neural networks
Polikar, R., Upda, L., Upda, S. S., & Honavar, V. (2001) · IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/5326.983933 · Cited by 888 · see citing works
Catastrophic forgetting, rehearsal and pseudorehearsal
ROBINS, A. (1995) · Connection Science · doi:10.1080/09540099550039318 · Cited by 833 · see citing works
Li, Z., & Hoiem, D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_37 · Cited by 659 · see citing works
Herding dynamical weights to learn
Welling, M. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553517 · Cited by 401 · see citing works
Metric Learning for Large Scale Image Classification: Generalizing to New Classes at Near-Zero Cost
Mensink, T., Verbeek, J., Perronnin, F., & Csurka, G. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33709-3_35 · Cited by 281 · see citing works
Muhlbaier, M. D., Topalis, A., & Robi Polikar, A. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2008326 · Cited by 209 · see citing works
Less-forgetting Learning in Deep Neural NetworksPreprint
Jung, H., Ju, J., Jung, M., & Kim, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.00122 · Cited by 160 · see citing works
Incremental Learning of Random Forests for Large-Scale Image Classification
Ristin, M., Guillaumin, M., Gall, J., & Gool, L. V. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2459678 · Cited by 152 · see citing works
Srivastava, R. K., Masci, J., Kazerounian, S., Gomez, F. J., & Schmidhuber, J. (2013) · neural information processing systems · Cited by 149 · see citing works
From N to N+1: Multiclass Transfer Incremental Learning
Kuzborskij, I., Orabona, F., & Barbara Caputo, B. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.431 · Cited by 139 · see citing works
Pseudo-recurrent Connectionist Networks: An Approach to the 'Sensitivity-Stability' Dilemma
French, R. M. (1997) · Connection Science · doi:10.1080/095400997116595 · Cited by 116 · see citing works
Continual Learning in Generative Adversarial NetsPreprint
Seff, A., Beatson, A., Suo, D., & Liu, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08395 · Cited by 88 · see citing works
Catastrophic interference in connectionist networks: Can It Be predicted, can It be prevented?
French, R. M. (1993) · neural information processing systems · Cited by 27 · see citing works
Active Long Term Memory NetworksPreprint
Furlanello, T., Zhao, J., Saxe, A., Itti, L., & Tjan, B. S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.02355 · Cited by 22 · see citing works
Jantke, K. P. (2002) · Cited by 15 · see citing works
Diving deeper into mentee networks
Venkatesan, R., & Li, B. (2016) · arXiv · doi:10.48550/arxiv.1604.08220 · Cited by 14 · see citing works
Self-citationOn the generality of neural image features
Venkatesan, R., Gatupalli, V., & Li, B. (2016) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2016.7532315 · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
1 of the 29 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Large Scale Incremental Learning
Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., & Fu, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00046 · Cited by 1,313 · see citing works
Three scenarios for continual learningPreprint
van de Ven, G. M., & Tolias, A. S. (2019) · Lirias · doi:10.48550/arxiv.1904.07734 · Cited by 714 · see citing works
Mnemonics Training: Multi-Class Incremental Learning Without Forgetting
LIU, Y., SU, Y., LIU, A.-A., Schiele, B., & SUN, Q. (2020) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 278 · see citing works
Domain Adaptation Without Source Data
Kim, Y., Cho, D., Han, K., Panda, P., & Hong, S. (2021) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3110179 · Cited by 200 · see citing works
Incremental Learning in Online Scenario
He, J., Mao, R., Shao, Z., & Zhu, F. M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01394 · Cited by 180 · see citing works
FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning
Petit, G., Popescu, A., Schindler, H., Picard, D., & Delezoide, B. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00390 · Cited by 171 · see citing works
Generative replay with feedback connections as a general strategy for continual learningPreprint
van de Ven, G. M., & Tolias, A. S. (2018) · Lirias · doi:10.48550/arxiv.1809.10635 · Cited by 194 · see citing works
An Appraisal of Incremental Learning Methods
Luo, Y., Yin, L., Bai, W., & Mao, K. (2020) · Entropy · doi:10.3390/e22111190 · Cited by 120 · see citing works
Continual Learning of a Mixed Sequence of Similar and Dissimilar TasksPreprint
Ke, Z., Liu, B., & Huang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.10017 · Cited by 73 · see citing works
OpenLORIS-Object: A Robotic Vision Dataset and Benchmark for Lifelong Deep Learning
She, Q., Feng, F., Hao, X., Yang, Q., Lan, C., Lomonaco, V., Shi, X., Wang, Z., Guo, Y., Zhang, Y., et al. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196887 · Cited by 69 · see citing works
Continual Learning: A Review of Techniques, Challenges, and Future Directions
Wickramasinghe, B., Saha, G., & Roy, K. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3339091 · Cited by 73 · see citing works
Revisiting Distillation and Incremental Classifier Learning
Javed, K., & Shafait, F. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-20876-9_1 · Cited by 53 · see citing works
Learning Without Forgetting: A New Framework for Network Cyber Security Threat Detection
Karn, R. R., Kudva, P. N., & Elfadel, I. M. (2021) · IEEE Access · doi:10.1109/access.2021.3115946 · Cited by 20 · see citing works
Online Continual Learning Via Candidates Voting
He, J., & Zhu, F. M. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00136 · Cited by 20 · see citing works
Ye, X., & Zhu, Q. (2019) · IEEE Access · doi:10.1109/access.2019.2904614 · Cited by 18 · see citing works
Ye, J., Liu, S., & Wang, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01928 · Cited by 16 · see citing works
A convolutional fuzzy min-max neural network
Chavan, T. R., & Nandedkar, A. V. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.04.003 · Cited by 14 · see citing works
Generative Models from the perspective of Continual Learning
Lesort, T., Caselles-Dupré, H., Garcia-Ortiz, M., Stoian, A., & Filliat, D. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8851986 · Cited by 13 · see citing works
Distillation Techniques for Pseudo-rehearsal Based Incremental LearningPreprint
Shah, H., Javed, K., & Faisal Shafait, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.02799 · Cited by 11 · see citing works
FeTrIL: Feature Translation for Exemplar-Free Class-Incremental LearningPreprint
Petit, G., Popescu, A., Schindler, H., Picard, D., & Delezoide, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.13131 · Cited by 11 · see citing works
Batbaatar, E., Park, K. H., Amarbayasgalan, T., Davagdorj, K., Munkhdalai, L., Pham, V.-H., & Ryu, K. H. (2020) · IEEE Access · doi:10.1109/access.2020.3039624 · Cited by 10 · see citing works
Introduction to Domain Adaptation
Venkateswara, H., & Panchanathan, S. (2020) · Springer eBooks · doi:10.1007/978-3-030-45529-3_1 · Cited by 9 · see citing works
Self-citationOpen-Ended Online Learning for Autonomous Visual Perception
Yu, H., Cong, Y., Sun, G., Hou, D., Liu, Y., & Dong, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3242448 · Cited by 8 · see citing works
Leveraging joint incremental learning objective with data ensemble for class incremental learning
Mazumder, P., Karim, M. A., Joshi, I., & Singh, P. P. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.01.017 · Cited by 7 · see citing works
Continual Learning of Medical Image Classification Based on Feature Replay
Li, X., Li, H., & Ma, L. (2022) · International Conference on Signal Processing · doi:10.1109/icsp56322.2022.9965230 · Cited by 4 · see citing works
A comparative study of calibration methods for imbalanced class incremental learning
Aggarwal, U., Popescu, A., Belouadah, E., & Hudelot, C. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-020-10485-5 · Cited by 4 · see citing works
Incremental classifier learning based on PEDCC-loss and cosine distance
Zhu, Q., He, Z., & Ye, X. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11163-w · Cited by 3 · see citing works
Class-Incremental Learning Based on Big Dataset Pre-Trained Models
Wen, B., & Zhu, Q. (2023) · IEEE Access · doi:10.1109/access.2023.3287771 · Cited by 3 · see citing works
Online Training from Streaming Data with Concept Drift on FPGAs
Roorda, E., & Wilton, S. J. E. (2023) · International Symposium on Quality Electronic Design (ISQED) · doi:10.1109/isqed57927.2023.10129312 · Cited by 2 · see citing works
Learning to Remember from a Multi-Task TeacherPreprint
Xiong, Y., Ren, M., & Urtasun, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.04650 · Cited by 2 · see citing works
Face recognition by deep learning
Wu, Y. C. (2019) · doi:10.17760/d20324043 · Cited by 2 · see citing works
Dominant Parameter Identification for Virtual Load Model Based on Incremental Learning
Hu, X., Zhang, X., Zeng, Y., Feng, C., & Meng, D. (2020) · IEEE Conference on Energy Internet and Energy System Integration (EI2) · doi:10.1109/ei250167.2020.9346815 · Cited by 1 · see citing works
Generative Low-Shot Network Expansion
Hayat, A., Kliger, M., Fleishman, S., & Cohen‐Or, D. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594004
Generative Low-Shot Network ExpansionPreprint
Hayat, A., Kliger, M., Fleishman, S., & Cohen‐Or, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.08363
Better Knowledge Retention through Metric LearningPreprint
Li, K., Peng, S., Vodrahalli, K., & Malik, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13149
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
Belouadah, E., Popescu, A., & Kanellos, I. (2020) · arXiv · doi:10.48550/arxiv.2011.01844 · Cited by 235 · see citing works
“me as random selection, with higher computational time. 7.3. Incremental learning without memory In many applications, no memory of past classes is available. For instance, in medical data processing [72], this is often due to privacy issues. We study the behavior of the main algorithms which can be deployed in the ab…”
BackgroundInfluential
- Authors
- Venkatesan, R., Venkateswara, H., Panchanathan, S., & Li, B.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1705.00744
- OpenAlex
- W2610828655
- Cited by
- 37
- References
- 29
- Citations by year
- 2023: 7 · 2022: 3 · 2021: 5 · 2020: 10 · 2019: 8 · 2018: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Data Stream Mining Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- catastrophic forgetting, incremental learning, continual learning, multi-class classification, generative models, knowledge distillation, dark knowledge, hallucinations, cross-domain learning, deep neural network
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
