Learning without Forgetting
Abstract
When building a unified vision system or gradually adding new apabilities to a system, the usual assumption is that training data for all tasks is always available. However, as the number of tasks grows, storing and retraining on such data becomes infeasible. A new problem arises where we add new capabilities to a Convolutional Neural Network (CNN), but the training data for its existing capabilities are unavailable. We propose our Learning without Forgetting method, which uses only new task data to train the network while preserving the original capabilities. Our method performs favorably compared to commonly used feature extraction and fine-tuning adaption techniques and performs similarly to multitask learning that uses original task data we assume unavailable. A more surprising observation is that Learning without Forgetting may be able to replace fine-tuning with similar old and new task datasets for improved new task performance.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LwF has even lower AP than direct fine-tuning in the detection task.
“Table 1 shows that although LwF works well in incremental classification, it has even lower AP than direct fine-tuning in detection task”
From Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation (2022) · cites this paper as [21]
DBLL excels in efficiently preserving knowledge through selective experience replay without needing complex architectural changes or parameter adjustments.
“Compared to ABLL and RBLL, Data-Based Lifelong Learning (DBLL) excels in efficiently preserving knowledge through selective experience replay [17,18] without needing complex architectural changes or parameter adjustments.”
From GPT-based lifelong learning and ANFIS-driven reply memory ratio prediction for aspect-based sentiment analysis (2025) · cites this paper as [18]
Using learning without forgetting, the model was fine-tuned on nerve-specific data while regularizing its outputs to general vessel-related structural knowledge, thereby reducing the risk of catastrophic forgetting.
“Using learning without forgetting, the model was fine-tuned on nerve-specific data while regularizing its outputs to general vessel-related structural knowledge represented in the pretrained weights, thereby reducing the risk of catastrophic forgetting49.”
From A deep-learning framework reveals whole-body perturbations at cell level (2026) · cites this paper as [48]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · see citing works
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899 · see citing works
The Pascal Visual Object Classes Challenge: A Retrospective
Everingham, M., Eslami, S. M. A., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-014-0733-5 · Cited by 7,139 · see citing works
Caruana, R. (1997) · Machine Learning · doi:10.1023/a:1007379606734 · Cited by 6,636 · see citing works
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · see citing works
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019 · see citing works
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770 · see citing works
Growing a Brain: Fine-Tuning by Increasing Model Capacity
Wang, Y.-X., Ramanan, D., & Hebert, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.323 · Cited by 154 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
iCaRL: Incremental Classifier and Representation Learning
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., & Lampert, C. H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.587 · Cited by 3,988 · see citing works
Knowledge Distillation: A Survey
Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01453-z · Cited by 3,931 · see citing works
How to backdoor federated learning
Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., & Shmatikov, V. (2018) · arXiv · doi:10.48550/arxiv.1807.00459 · Cited by 1,495 · see citing works
Dota 2 with Large Scale Deep Reinforcement Learning
OpenAI, :, Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.06680 · Cited by 1,065 · see citing works
A Comprehensive Survey of Continual Learning: Theory, Method and Application
Wang, L., Zhang, X., Su, H., & Zhu, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3367329 · Cited by 1,113 · see citing works
Three types of incremental learning
van de Ven, G. M., Tuytelaars, T., & Tolias, A. S. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00568-3 · Cited by 665 · see citing works
Incremental Learning of Object Detectors without Catastrophic Forgetting
Shmelkov, K., Schmid, C., & Alahari, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.368 · Cited by 559 · see citing works
Shen, S., Sadoughi, M., Li, M., Wang, Z., & Hu, C. (2020) · Applied Energy · doi:10.1016/j.apenergy.2019.114296 · Cited by 537 · see citing works
SpotTune: Transfer Learning Through Adaptive Fine-Tuning
Guo, Y., Shi, H., Kumar, A., Grauman, K., Rosing, T., & Feris, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00494 · Cited by 506 · see citing works
Brain-inspired replay for continual learning with artificial neural networks
van de Ven, G. M., Siegelmann, H. T., & Tolias, A. S. (2020) · Nature Communications · doi:10.1038/s41467-020-17866-2 · Cited by 474 · see citing works
Prototype Augmentation and Self-Supervision for Incremental Learning
Zhu, F., Zhang, X.-Y., Wang, C., Yin, F., & Liu, C.-L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00581 · Cited by 427 · see citing works
Continual learning of context-dependent processing in neural networks
Zeng, G., Chen, Y., Cui, B., & Yu, S. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0080-x · Cited by 309 · see citing works
Tree-CNN: A hierarchical Deep Convolutional Neural Network for incremental learning
Roy, D., Panda, P., & Roy, K. (2020) · Neural Networks · doi:10.1016/j.neunet.2019.09.010 · Cited by 257 · see citing works
Federated learning improves site performance in multicenter deep learning without data sharing
Sarma, K. V., Harmon, S., Sanford, T., Roth, H. R., Xu, Z., Tetreault, J., Xu, D., Flores, M. G., Raman, A. G., Kulkarni, R., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa341 · Cited by 242 · see citing works
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
Belouadah, E., Popescu, A., & Kanellos, I. (2020) · arXiv · doi:10.48550/arxiv.2011.01844 · Cited by 235 · see citing works
Knowledge Distillation with the Reused Teacher Classifier
Chen, D., Mei, J.-P., Zhang, H., Wang, C., Feng, Y., & Chen, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01163 · Cited by 254 · see citing works
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224 · see citing works
Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
Hsu, Y.-C., Liu, Y.-C., Ramasamy, A., & Kira, Z. (2018) · arXiv · doi:10.48550/arxiv.1810.12488 · Cited by 211 · see citing works
Lifelong Learning via Progressive Distillation and Retrospection
Hou, S., Pan, X., Loy, C. C., Wang, Z., & Lin, D. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_27 · Cited by 190 · see citing works
Pan-Peptide Meta Learning for T-cell receptor–antigen binding recognition
Gao, Y., Gao, Y., Fan, Y., Zhu, C., Wei, Z., Zhou, C., Chuai, G., Chen, Q., Zhang, H., & Liu, Q. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00619-3 · Cited by 155 · see citing works
Topology-Preserving Class-Incremental Learning
Tao, X., Chang, X., Hong, X., Wei, X., & Gong, Y. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58529-7_16 · Cited by 162 · see citing works
An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-Tuning
Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., & Zhang, Y. (2025) · IEEE Transactions on Audio, Speech and Language Processing · doi:10.1109/taslpro.2025.3606231 · Cited by 136 · see citing works
An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2308.08747 · Cited by 135 · see citing works
Zheng, S., Pan, K., Liu, J., & Chen, Y. (2024) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2024.110382 · Cited by 127 · see citing works
Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation
Feng, T., Wang, M., & Yuan, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00921 · Cited by 132 · see citing works
An Appraisal of Incremental Learning Methods
Luo, Y., Yin, L., Bai, W., & Mao, K. (2020) · Entropy · doi:10.3390/e22111190 · Cited by 120 · see citing works
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Räuker, T., Ho, A., Casper, S., & Hadfield-Menell, D. (2023) · 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00039 · Cited by 122 · see citing works
Dynamic Memory to Alleviate Catastrophic Forgetting in Continual Learning with Medical Imaging
Perkonigg, M., Hofmanninger, J., Herold, C. J., Brink, J. A., Pianykh, O., Prosch, H., & Langs, G. (2021) · Nature Communications · doi:10.1038/s41467-021-25858-z · Cited by 106 · see citing works
Continual Relation Learning via Episodic Memory Activation and Reconsolidation
Han, X., Dai, Y., Gao, T., Lin, Y., Liu, Z., Li, P., Sun, M., & Zhou, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.573 · Cited by 104 · see citing works
Li, D., Tasci, S., Ghosh, S., Zhu, J., Zhang, J., & Heck, L. (2019) · Proceedings of the 4th ACM/IEEE Symposium on Edge Computing · doi:10.1145/3318216.3363317 · Cited by 94 · see citing works
An End-to-End Architecture for Class-Incremental Object Detection with Knowledge Distillation
Hao, Y., Fu, Y., Jiang, Y.-G., & Tian, Q. (2019) · 2019 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2019.00009 · Cited by 98 · see citing works
Thakur, S., Doshi, J., Pati, S., Rathore, S., Sako, C., Bilello, M., Ha, S. M., Shukla, G., Flanders, A., Kotrotsou, A., et al. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117081 · Cited by 84 · see citing works
Overcoming Multi-Model Forgetting in One-Shot NAS With Diversity Maximization
Zhang, M., Li, H., Pan, S., Chang, X., & Su, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00783 · Cited by 83 · see citing works
Orthogonal Subspace Learning for Language Model Continual Learning
Wang, X., Chen, T., Ge, Q., Xia, H., Bao, R., Zheng, R., Zhang, Q., Gui, T., & Huang, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.715 · Cited by 79 · see citing works
Federated Learning Empowered Real-Time Medical Data Processing Method for Smart Healthcare
Guo, K., Chen, T., Ren, S., Li, N., Hu, M., & Kang, J. (2024) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2022.3185395 · Cited by 63 · see citing works
An Intelligent Transient Stability Assessment Framework with Continual Learning Ability
Li, X., Yang, Z., Guo, P., & Cheng, J. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2021.3064052 · Cited by 62 · see citing works
Robustifying the Deployment of tinyML Models for Autonomous mini-vehicles
de Prado, M., Rusci, M., Capotondi, A., Donze, R., Benini, L., & Pazos, N. (2021) · Sensors · doi:10.3390/s21041339 · Cited by 57 · see citing works
Huang, T., Chen, P., & Li, R. (2022) · Proceedings of the ACM Web Conference 2022 · doi:10.1145/3485447.3511984 · Cited by 61 · see citing works
Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
Aleixo, E. L., Colonna, J. G., Cristo, M., & Fernandes, E. (2023) · arXiv · doi:10.48550/arxiv.2312.10549 · Cited by 54
Online Learning of Unknown Dynamics for Model-Based Controllers in Legged Locomotion
Sun, Y., Ubellacker, W. L., Ma, W.-L., Zhang, X., Wang, C., Csomay-Shanklin, N. V., Tomizuka, M., Sreenath, K., & Ames, A. D. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3108510 · Cited by 54 · see citing works
Industrial Transfer Learning: Boosting Machine Learning in Production
Tercan, H., Guajardo, A., & Meisen, T. (2019) · 2019 IEEE 17th International Conference on Industrial Informatics (INDIN) · doi:10.1109/indin41052.2019.8972099 · Cited by 50 · see citing works
Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
Aleixo, E. L., Colonna, J. G., Cristo, M., & Fernandes, E. (2024) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2024.3966 · Cited by 54 · see citing works
Challenges in Task Incremental Learning for Assistive Robotics
Feng, F., Chan, R. H. M., Shi, X., Zhang, Y., & She, Q. (2020) · IEEE Access · doi:10.1109/access.2019.2955480 · Cited by 39 · see citing works
Knowledge from the Original Network: Restore a Better Pruned Network with Knowledge Distillation
Chen, L., Chen, Y., Xi, J., & Le, X. (2022) · Complex & Intelligent Systems · doi:10.1007/s40747-020-00248-y · Cited by 48 · see citing works
Learning with Selective Forgetting
Shibata, T., Irie, G., Ikami, D., & Mitsuzumi, Y. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/137 · Cited by 47 · see citing works
Continual Learning for Fake Audio Detection
Ma, H., Yi, J., Tao, J., Bai, Y., Tian, Z., & Wang, C. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-794 · Cited by 39 · see citing works
DFIL: Deepfake Incremental Learning by Exploiting Domain-invariant Forgery Clues
Pan, K., Yin, Y., Wei, Y., Lin, F., Ba, Z., Liu, Z., Wang, Z., Cavallaro, L., & Ren, K. (2023) · Proceedings of the 31st ACM International Conference on Multimedia · doi:10.1145/3581783.3612377 · Cited by 37 · see citing works
What to remember: Self-adaptive continual learning for audio deepfake detection
Zhang, X., Yi, J., Wang, C., Zhang, C. Y., Zeng, S., & Tao, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29929 · Cited by 32 · see citing works
Adversarial Feature Alignment: Avoid Catastrophic Forgetting in Incremental Task Lifelong Learning
Yao, X., Huang, T., Wu, C., Zhang, R.-X., & Sun, L. (2019) · Neural Computation · doi:10.1162/neco_a_01232 · Cited by 27 · see citing works
Class-incremental learning network for real-time anomaly recognition in surveillance environments
Hussain, A., Ullah, W., Khan, N., Khan, Z. A., Yar, H., & Baik, S. W. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2025.112064 · Cited by 24 · see citing works
Continual learning of longitudinal health records
Armstrong, J., & Clifton, D. A. (2022) · 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI) · doi:10.1109/bhi56158.2022.9926878 · Cited by 21 · see citing works
Open-world continual learning: Unifying novelty detection and continual learning
Kim, G., Xiao, C., Konishi, T., Ke, Z., & Liu, B. (2025) · Artificial Intelligence · doi:10.1016/j.artint.2024.104237 · Cited by 19 · see citing works
Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
Igl, M., Farquhar, G., Luketina, J., Boehmer, W., & Whiteson, S. (2020) · arXiv · doi:10.48550/arxiv.2006.05826 · Cited by 17 · see citing works
Self-Net: Lifelong Learning via Continual Self-Modeling
Mandivarapu, J. K., Camp, B., & Estrada, R. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.00019 · Cited by 14 · see citing works
Learning With Sharing: An Edge-Optimized Incremental Learning Method for Deep Neural Networks
Hussain, M. A., Huang, S.-A., & Tsai, T.-H. (2023) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2022.3210905 · Cited by 11 · see citing works
Continual Egocentric Activity Recognition With Foreseeable-Generalized Visual–IMU Representations
He, C., Cheng, S., Qiu, Z., Xu, L., Meng, F., Wu, Q., & Li, H. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3371975 · Cited by 11 · see citing works
Revisiting Flatness-Aware Optimization in Continual Learning With Orthogonal Gradient Projection
Yang, E., Shen, L., Wang, Z., Liu, S., Guo, G., Wang, X., & Tao, D. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3539019 · Cited by 5 · see citing works
YOLO-IOD: Towards Real Time Incremental Object Detection
Zhang, S., Lv, X., Xing, Y., Wu, Q., Xu, D., Zhao, C., & Zhang, Y. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i15.38271 · Cited by 4 · see citing works
Advancing Continual Learning for Robust Deepfake Audio Classification
Dong, F., Tang, Q., Bai, Y., & Wang, Z. (2024) · TENCON 2024 - 2024 IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon61640.2024.10902761 · Cited by 3 · see citing works
A deep-learning framework reveals whole-body perturbations at cell level
Kaltenecker, D., Horvath, I., Al-Maskari, R., Chen, Y., Kolabas, Z. I., Hoeher, L., Todorov, M., Minde, D.-P., Kapoor, S., Turhan, S. G., et al. (2026) · Nature · doi:10.1038/s41586-026-10535-2 · Cited by 5 · see citing works
Huang, H., Ye, X., Mustafa, M. B., Dong, Q., Li, Y., Asemi, A., & Asemi, A. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-02086-2
- Authors
- Li, Z., & Hoiem, D.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2018-12-01
- Volume
- 40
- Issue
- 12
- Pages
- 2935-2947
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2473930607
- Cited by
- 4,202
- References
- 10
- Citations by year
- 2026: 596 · 2025: 747 · 2024: 729 · 2023: 654 · 2022: 485 · 2021: 434 · 2020: 311 · 2019: 165 · 2018: 64 · 2017: 14 · 2016: 3
- FWCI
- 90.51
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- Funders
- NSF (14-46765, 10-53768)
- ONR MURI (N000014-16-1-2007)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- learning without forgetting, catastrophic forgetting, convolutional neural networks, continual learning, fine-tuning, multi-task learning, feature extraction, vision system
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
