A Comprehensive Survey of Continual Learning: Theory, Method and Application
Open accessThe DOI matches its Crossref record; 456 of the 456 references it cites were found in a registry or an index.
Abstract
To cope with real-world dynamics, an intelligent system needs to incrementally acquire, update, accumulate, and exploit knowledge throughout its lifetime. This ability, known as continual learning, provides a foundation for AI systems to develop themselves adaptively. In a general sense, continual learning is explicitly limited by catastrophic forgetting, where learning a new task usually results in a dramatic performance drop of the old tasks. Beyond this, increasingly numerous advances have emerged in recent years that largely extend the understanding and application of continual learning. The growing and widespread interest in this direction demonstrates its realistic significance as well as complexity. In this work, we present a comprehensive survey of continual learning, seeking to bridge the basic settings, theoretical foundations, representative methods, and practical applications. Based on existing theoretical and empirical results, we summarize the general objectives of continual learning as ensuring a proper stability-plasticity trade-off and an adequate intra/inter-task generalizability in the context of resource efficiency. Then we provide a state-of-the-art and elaborated taxonomy, extensively analyzing how representative strategies address continual learning, and how they are adapted to particular challenges in various applications. Through an in-depth discussion of promising directions, we believe that such a holistic perspective can greatly facilitate subsequent exploration in this field and beyond.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Complex skills can be synthesized by composing simpler programs, which compounds VOYAGER's capabilities rapidly over time and alleviates catastrophic forgetting in other continual learning methods [31, 32].”
From Voyager: An Open-Ended Embodied Agent with Large Language Models (2023) · cites this paper as [32]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, these agents are not lifelong learners that can progressively acquire, update, accumulate, and transfer knowledge over extended time spans [31, 32].”
From Voyager: An Open-Ended Embodied Agent with Large Language Models (2023) · cites this paper as [32]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This position also makes memory currently the most practical route to continual learning for deployed agents: retraining the underlying model for every user, task, and correction is rarely feasible, so what the system learns after deployment often depends on what it writes down [26, 67].”
From Agentic Memory Management Is the Next Frontier of Information Retrieval (2026) · cites this paper as [67]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In AI, memory is usually described in terms of persisting state, retaining conversational context, or learning continually without catastrophic forgetting [32, 67].”
From Agentic Memory Management Is the Next Frontier of Information Retrieval (2026) · cites this paper as [67]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The stability–plasticity dilemma is central to CL, as it balances the model’s ability to retain past knowledge (stability) while acquiring new information (plasticity) [61,62].”
From Deepfake Media Forensics: Status and Future Challenges (2025) · cites this paper as [62]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“Transformer-Based Architecture [317] has proven effective for both language and vision domains, and become the dominant choice for state-of-the-art foundation models.”
BackgroundExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · cited 1×
“According to the learning theory for supervised learning [48], the performance of an old task is improved with replaying more old training samples that approximate its distribution, but resulting in potential privacy issues and a linear increase in resource overhead .”
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“Foundation Model , such as GPT [316] and CLIP [218], demonstrates impressive performance in a variety of down-stream tasks with the use of large-scale pre-training.”
MethodGAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 4×
“erally more efficient than images, IncCLIP [467] replays generated hard negative texts conditioned on images and performs multi-modal knowledge distillation for updating CLIP [347].”
BackgroundMethodInfluentialCommunication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512 · cited 1×
“of synaptic plasticity” [3], [4], [77], corresponding to the use of meta-learning [118], [185], [360]; and (4) inhibitory synapses for excited neurons to reduce the activity of other neurons [20], [54], [112], [178], which acts similarly to the binary mask for parameter allocation [375], [427].”
Method
Research works printed but not linked: 4
- Ramasesh, Effect of scale on catastrophic forgetting in neural networks
- Benzing, Unifying importance based regularisation methods for continual learning, 2372
- Miao, Continual learning with filter atom swapping
- Ardywibowo, VariGrow: Variational architecture growing for task-agnostic continual learning based on Bayesian novelty, 865
Cited by
Atakishiyev, S., Salameh, M., Yao, H., & Goebel, R. (2024) · IEEE Access · doi:10.1109/access.2024.3431437 · Cited by 323
Atakishiyev, S., Salameh, M., Yao, H., & Goebel, R. (2021) · arXiv · doi:10.48550/arxiv.2112.11561 · Cited by 336
Vision-language models for medical report generation and visual question answering: a review
Hartsock, I., & Rasool, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1430984 · Cited by 229
A Comprehensive Survey of Convolutions in Deep Learning: Applications, Challenges, and Future Trends
Younesi, A., Ansari, M., Fazli, M., Ejlali, A., Shafique, M., & Henkel, J. (2024) · IEEE Access · doi:10.1109/access.2024.3376441 · Cited by 197
“Models must avoid catastrophic forgetting when presented with new classes or shifts in existing class definitions without revisiting historical data [391], [392], [393], [394], [395], [396], [397], [398], [399], [400].”
BackgroundVoyager: An Open-Ended Embodied Agent with Large Language Models
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023) · arXiv · doi:10.48550/arxiv.2305.16291 · Cited by 222
“[32] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu.”
Background
Metadata
- Authors
- Wang, L., Zhang, X., Su, H., & Zhu, J.
- Affiliations
- Tsinghua University (CN)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2024-08
- Volume
- 46
- Issue
- 8
- Pages
- 5362-5383
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4392173735
- Cited by
- 1,113
- References
- 313
- Citations by year
- 2026: 437 · 2025: 471 · 2024: 170
- FWCI
- 158.75
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62350080, 62076147, 62106123, U19A2081)
- BNRist (BNR2022RC01006)
- Tsinghua Institute for Guo Qiang
- Tsinghua-OPPO Joint Research Center for Future Terminal Technology
- High Performance Computing Center
- Tsinghua University
- Postdoctoral Fellowship Program of CPSF (GZB20230350)
- Shuimu Tsinghua Scholar
- New Cornerstone Science Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Data Stream Mining Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- continual learning, catastrophic forgetting, stability-plasticity trade-off, resource efficiency, incremental learning, adaptive intelligent systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Comprehensive Survey of Continual Learning: Theory, Method and Application. https://app.cochat.ai/papers/10.1109/tpami.2024.3367329
- The paper
- Wang, L., Zhang, X., Su, H., & Zhu, J. (2024). A Comprehensive Survey of Continual Learning: Theory, Method and Application. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2024.3367329
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.