CoChat
Sign Up
PreprintCited by 135Domain Adaptation and Few-Shot LearningTopic ModelingNatural Language Processing Techniques

The DOI matches its DataCite record.

Continual Learning of Large Language Models: A Comprehensive Survey

Open access
Shi, Haizhou, Xu, Zihao, Wang, Hengyi, Qin, Weiyi, Wang, Wenyuan, Wang, Yibin, Wang, Zifeng, Ebrahimi, Sayna, Wang, Hao·2024·arXiv
DOI 10.48550/arxiv.2404.16789arXiv 2404.16789
Open this paper in CoChat — free accountOpen PDF

Abstract

The recent success of large language models (LLMs) trained on static, pre-collected, general datasets has sparked numerous research directions and applications. One such direction addresses the non-trivial challenge of integrating pre-trained LLMs into dynamic data distributions, task structures, and user preferences. Pre-trained LLMs, when tailored for specific needs, often experience significant performance degradation in previous knowledge domains -- a phenomenon known as "catastrophic forgetting". While extensively studied in the continual learning (CL) community, it presents new manifestations in the realm of LLMs. In this survey, we provide a comprehensive overview of the current research progress on LLMs within the context of CL. This survey is structured into four main sections: we first describe an overview of continually learning LLMs, consisting of two directions of continuity: vertical continuity (or vertical continual learning), i.e., continual adaptation from general to specific capabilities, and horizontal continuity (or horizontal continual learning), i.e., continual adaptation across time and domains (Section 3). We then summarize three stages of learning LLMs in the context of modern CL: Continual Pre-Training (CPT), Domain-Adaptive Pre-training (DAP), and Continual Fine-Tuning (CFT) (Section 4). Then we provide an overview of evaluation protocols for continual learning with LLMs, along with the current available data sources (Section 5). Finally, we discuss intriguing questions pertaining to continual learning for LLMs (Section 6). The full list of papers examined in this survey is available at https://github.com/Wang-ML-Lab/llm-continual-learning-survey.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2404.16789

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 300

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv:2303.08774. Retrieved from https://arxiv.org/abs/2303.08774 · arXiv
  2. Emre Can Acikgoz, Osman Batur İnce, Rayene Bench, Arda Anıl Boz, İlker Kesen, Aykut Erdem, and Erkut Erdem. 2024. Hippocrates: An open-source framework for advancing large language models in healthcare. arXiv:2404.16621. Retrieved from https://arxiv.org/abs/2404.16621 · arXiv
  3. Mayank Agarwal, Yikang Shen, Bailin Wang, Yoon Kim, and Jie Chen. 2024. Structured Code Representations Enable Data-Efficient Adaptation of Code Language Models. arXiv:2401.10716. Retrieved from https://arxiv.org/abs/2401. 10716 · arXiv
  4. Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV). 139-154.
  5. Spurthi Amba Hombaiah, Tao Chen, Mingyang Zhang, Michael Bendersky, and Marc Najork. 2021. Dynamic lan- guage models for continuously evolving content. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2514-2524.
  6. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv:2305.10403. Retrieved from https://arxiv.org/abs/2305.10403 · arXiv
  7. Dogu Araci. 2019. FinBERT: Financial Sentiment Analysis with Pre-trained Language Models. arXiv:1908.10063. Re- trieved from https://arxiv.org/abs/1908.10063 · arXiv
  8. Giuseppe Attanasio, Debora Nozza, Federico Bianchi, and Dirk Hovy. 2023. Is It Worth the (Environmental) Cost? Limited Evidence for Temporal Adaptation via Continuous Training. arXiv:2210.07365. Retrieved from https://arxiv. org/abs/2210.07365 · arXiv
  9. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2023. Llemma: An open language model for mathematics. arXiv:2310.10631. Re- trieved from https://arxiv.org/abs/2310.10631 · arXiv
  10. Xueying Bai, Jinghuan Shang, Yifan Sun, and Niranjan Balasubramanian. 2023. Enhancing Continual Learning with Global Prototypes: Counteracting Negative Representation Drift. arXiv:2205.12186. Retrieved from https://arxiv.org/ abs/2205.12186 · arXiv
  11. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862. Retrieved from https://arxiv.org/abs/2204.05862 · arXiv
  12. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correla- tion with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. 65-72.
  13. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The Pushshift Reddit Dataset. arXiv:2001.08435. Retrieved from https://arxiv.org/abs/2001.08435 · arXiv
  14. Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010. A theory of learning from different domains. Machine Learning 79 (2010), 151-175.
  15. Zhen Bi, Ningyu Zhang, Yida Xue, Yixin Ou, Daxiong Ji, Guozhou Zheng, and Huajun Chen. 2023. OceanGPT: A large language model for ocean science tasks. arXiv:2310.02031. Retrieved from https://arxiv.org/abs/2310.02031 · arXiv
  16. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Moham- mad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the International Conference on Machine Learning. PMLR, 2397-2430.
  17. Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. 2020. Continual lifelong learning in natural language processing: A survey. In Proceedings of the 28th International Conference on Computational Linguistics, Donia Scott, Nuria Bel, and Chengqing Zong (Eds.). International Committee on Computational Linguistics, Barcelona, Spain (Online), 6523-6541. DOI:https://doi.org/10.18653/v1/2020.coling-main.574 · DOI
  18. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and others. 2020. Piqa: Reasoning about physical common- sense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence. 7432-7439.
  19. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the 9th Workshop on Statistical Machine Translation. 12-58.
  20. Jorg Bornschein, Yazhe Li, and Amal Rannen-Triki. 2024. Transformers for Supervised Online Continual Learning. arXiv:2403.01554. Retrieved from https://arxiv.org/abs/2403.01554 · arXiv
  21. Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2020. Machine Unlearning. arXiv:1912.03817. Retrieved from https://arxiv.org/abs/ 1912.03817 · arXiv
  22. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020), 1877-1901.
  23. Martin Brümmer, Milan Dojchinovski, and Sebastian Hellmann. 2016. Dbpedia abstracts: A large-scale, open, multi- lingual NLP training corpus. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC'16). 3339-3343.
  24. Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. 2020. Dark experience for general continual learning: A strong, simple baseline. Advances in Neural Information Processing Systems 33 (2020), 15920-15930.
  25. He Cao, Zijing Liu, Xingyu Lu, Yuan Yao, and Yu Li. 2023. InstructMol: Multi-modal integration for building a versatile and reliable molecular assistant in drug discovery. arXiv:2311.16208. Retrieved from https://arxiv.org/abs/2311.16208 · arXiv
  26. Xusheng Cao, Haori Lu, Linlan Huang, Xialei Liu, and Ming-Ming Cheng. 2024. Generative multi-modal models are good class incremental learners. IEEE Computer Vision and Pattern Recognition (2024).
  27. Ilias Chalkidis, Tommaso Pasini, Sheng Zhang, Letizia Tomada, Sebastian Felix Schwemer, and Anders Søgaard. 2022. Fairlex: A multilingual benchmark for evaluating fairness in legal text processing. arXiv:2203.07228. Retrieved from https://arxiv.org/abs/2203.07228 · arXiv
  28. Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019. Efficient lifelong learning with a-GEM. In Proceedings of the ICLR.
  29. Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip HS Torr, and Marc'Aurelio Ranzato. 2019. On tiny episodic memories in continual learning. arXiv:1902.10486. Retrieved from https://arxiv.org/abs/1902.10486 · arXiv
  30. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2014. One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. arXiv:1312.3005. Retrieved from https://arxiv.org/abs/1312.3005 · arXiv
  31. Cheng Chen, Junchen Zhu, Xu Luo, Hengtao Shen, Lianli Gao, and Jingkuan Song. 2024. CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model. arXiv:2403.08350. Retrieved from https://arxiv. org/abs/2403.08350 · arXiv
  32. Junying Chen, Xidong Wang, Anningzhe Gao, Feng Jiang, Shunian Chen, Hongbo Zhang, Dingjie Song, Wenya Xie, Chuyi Kong, Jianquan Li, Xiang Wan, Haizhou Li, and Benyou Wang. 2023. HuatuoGPT-II, one-stage training for medical adaption of LLMs. arXiv:2311.09774. Retrieved from https://arxiv.org/abs/2311.09774 · arXiv
  33. Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020. Recall and learn: Fine- tuning deep pretrained language models with less forgetting. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). As- sociation for Computational Linguistics, Online, 7870-7881. DOI:https://doi.org/10.18653/v1/2020.emnlp-main.634 · DOI
  34. Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cui. 2023. Lifelong language pretraining with distribution-specialized experts. In Proceedings of the International Conference on Machine Learning. PMLR, 5383-5395.
  35. Xuxi Chen, Zhendong Wang, Daouda Sow, Junjie Yang, Tianlong Chen, Yingbin Liang, Mingyuan Zhou, and Zhangyang Wang. 2024. Take the bull by the horns: Hard sample-reweighted continual training improves LLM generalization. arXiv:2402.14270. Retrieved from https://arxiv.org/abs/2402.14270 · arXiv
  36. Yongrui Chen, Shenyu Zhang, Guilin Qi, and Xinnan Guo. 2024. Parameterizing context: Unleashing the power of parameter-efficient fine-tuning and in-context tuning for continual table semantic parsing. Advances in Neural Information Processing Systems 36 (2024).
  37. Zhiyuan Chen and Bing Liu. [n. d.]. Lifelong Machine Learning. Springer.
  38. Daixuan Cheng, Shaohan Huang, and Furu Wei. 2024. Adapting Large Language Models via Reading Comprehension. arXiv:2309.09530. Retrieved from https://arxiv.org/abs/2309.09530 · arXiv
  39. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with path- ways. Journal of Machine Learning Research 24, 240 (2023), 1-113.
  40. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? Try arc, the ai2 reasoning challenge. arXiv:1803.05457. Retrieved from https://arxiv.org/abs/1803.05457 · arXiv
  41. Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre F. T. Martins, Fab- rizio Esposito, Vera Lúcia Raposo, Sofia Morgado, and Michael Desa. 2024. SaulLM-7B: A pioneering Large Language Model for Law. arXiv:2403.03883. Retrieved from https://arxiv.org/abs/2403.03883 · arXiv
  42. Andrea Cossu, Tinne Tuytelaars, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco, and Davide Bacciu. 2022. Con- tinual Pre-Training Mitigates Forgetting in Language and Vision. arXiv:2205.09357. Retrieved from https://arxiv.org/ abs/2205.09357 · arXiv
  43. Payel Das, Subhajit Chaudhury, Elliot Nelson, Igor Melnyk, Sarath Swaminathan, Sihui Dai, Aurélie Lozano, Georgios Kollias, Vijil Chenthamarakshan, Soham Dan, et al. 2024. Larimar: Large language models with episodic memory control. arXiv:2403.11901. Retrieved from https://arxiv.org/abs/2403.11901 · arXiv
  44. Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. 2019. Quoref: A reading compre- hension dataset with questions requiring coreferential reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Lin- guistics, Hong Kong, China, 5925-5932. DOI:https://doi.org/10.18653/v1/D19-1606 · DOI
  45. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. arXiv:2104.08164. Retrieved from https://arxiv.org/abs/2104.08164 · arXiv
  46. Cheng Deng, Tianhang Zhang, Zhongmou He, Yi Xu, Qiyuan Chen, Yuanyuan Shi, Luoyi Fu, Weinan Zhang, Xin- bing Wang, Chenghu Zhou, Zhouhan Lin, and Junxian He. 2023. K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilization. arXiv:2306.05064. Retrieved from https://arxiv.org/abs/2306.05064 · arXiv
  47. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the CVPR09.
  48. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient finetuning of quan- tized LLMs. arXiv:2305.14314. Retrieved from https://arxiv.org/abs/2305.14314 · arXiv
  49. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. Retrieved from https://arxiv.org/abs/1810.04805 · arXiv
  50. Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics 10 (2022), 257-273.
  51. Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. 2022. Calibrating factual knowledge in pretrained language models. arXiv:2210.03329. Retrieved from https://arxiv.org/abs/2210.03329 · arXiv
  52. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929. Retrieved from https://arxiv.org/abs/2010.11929 · arXiv
  53. Longxu Dou, Qian Liu, Guangtao Zeng, Jia Guo, Jiahui Zhou, Wei Lu, and Min Lin. 2024. Sailor: Open language models for south-east asia. arXiv:2404.03608. Retrieved from https://arxiv.org/abs/2404.03608 · arXiv
  54. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv:1903.00161. Retrieved from https://arxiv.org/abs/1903.00161 · arXiv
  55. Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. 2019. Uncertainty-guided continual learning with bayesian neural networks. arXiv:1906.02425. Retrieved from https://arxiv.org/abs/1906.02425 · arXiv
  56. Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. 2020. Adversarial con- tinual learning. In Proceedings of the 16th European Conference on Computer Vision-ECCV 2020, Glasgow, UK, August 23-28, 2020, Part XI 16. Springer, 386-402.
  57. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  58. Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual pre-training for cross-lingual LLM adaptation: Enhancing japanese language capabilities. arXiv:2404.17790. Retrieved from https://arxiv.org/abs/2404.17790 · arXiv
  59. Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marc- hand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The Journal of Machine Learning Research 17, 1 (2016), 2096-2030.
  60. Saurabh Garg, Mehrdad Farajtabar, Hadi Pouransari, Raviteja Vemulapalli, Sachin Mehta, Oncel Tuzel, Vaishaal Shankar, and Fartash Faghri. 2024. TiC-CLIP: Continual training of CLIP models. In Proceedings of the 12th International Conference on Learning Representations (ICLR). Retrieved from https://openreview.net/forum?id= TLADT8Wrhn
  61. Evangelia Gogoulou, Timothée Lesort, Magnus Boman, and Joakim Nivre. 2024. Continual Learning Under Language Shift. arXiv:2311.01200. Retrieved from https://arxiv.org/abs/2311.01200 · arXiv
  62. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. arXiv:1612.00837. Retrieved from https: //arxiv.org/abs/1612.00837 · arXiv
  63. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare 3, 1 (2021), 1-23. DOI:https://doi.org/10.1145/3458754 · DOI · Paper page
  64. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming-The Rise of Code Intelligence. arXiv:2401.14196. Retrieved from https://arxiv.org/abs/2401.14196 · arXiv
  65. Zhen Guo and Yining Hua. 2023. Continuous Training and Fine-tuning for Domain-Specific Language Models in Medical Question Answering. arXiv:2311.00204. Retrieved from https://arxiv.org/abs/2311.00204 · arXiv
  66. Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats L. Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. 2023. Continual Pre-Training of Large Language Models: How to (re)warm your model? arXiv:2308.04014. Retrieved from https://arxiv.org/abs/2308.04014 · arXiv
  67. Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. 2018. VizWiz Grand Challenge: Answering Visual Questions from Blind People. arXiv:1802.08218. Retrieved from https://arxiv.org/abs/1802.08218 · arXiv
  68. Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2022. DEMix layers: Disentan- gling domains for modular language modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (Eds.). Association for Computational Linguistics, Seattle, United States, 5557-5576. DOI:https://doi.org/10.18653/v1/2022.naacl-main.407 · DOI
  69. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds.). Association for Computational Linguistics, Online, 8342-8360. DOI:https://doi.org/10.18653/v1/2020.acl-main.740 · DOI · Paper page
  70. Rujun Han, Xiang Ren, and Nanyun Peng. 2021. ECONET: Effective continual pretraining of language models for event temporal reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 5367-5380. DOI:https://doi.org/10.18653/v1/2021.emnlp- main.436 · DOI
  71. Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2023. Aging with GRACE: Lifelong model editing with discrete key-value adaptors. In Proceedings of the Advances in Neural Information Processing Systems.
  72. Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srini- vasan Iyer. 2021. Do language models have beliefs? Methods for detecting, updating, and visualizing model beliefs. arXiv:2111.13654. Retrieved from https://arxiv.org/abs/2111.13654 · arXiv
  73. Jinghan He, Haiyun Guo, Ming Tang, and Jinqiao Wang. 2023. Continual Instruction Tuning for Large Multimodal Models. arXiv:2311.16206. Retrieved from https://arxiv.org/abs/2311.16206 · arXiv
  74. Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one- class collaborative filtering. In Proceedings of the 25th International Conference on World Wide Web (WWW'16). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 507-517. DOI:https://doi.org/10.1145/2872427.2883037 · DOI
  75. Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, and Fuchun Peng. 2021. Analyzing the forgetting problem in pretrain-finetuning of open-domain dialogue response models. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (Eds.). Association for Computational Linguistics, Online, 1121-1133. DOI:https: //doi.org/10.18653/v1/2021.eacl-main.95 · DOI
  76. Yongquan He, Xuancheng Huang, Minghao Tang, Lingxun Meng, Xiang Li, Wei Lin, Wenyuan Zhang, and Yifu Gao. 2024. Don't Half-listen: Capturing Key-part Information in Continual Instruction Tuning. arXiv:2403.10056. Retrieved from https://arxiv.org/abs/2403.10056 · arXiv
  77. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2023. Align- ing AI With Shared Human Values. arXiv:2008.02275. Retrieved from https://arxiv.org/abs/2008.02275 · arXiv
  78. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Rep- resentations (2021).
  79. Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kel- cey, and David Berthelot. 2016. Wikireading: A novel large-scale language understanding task over wikipedia. arXiv:1608.03542. Retrieved from https://arxiv.org/abs/1608.03542 · arXiv
  80. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large lan- guage models. arXiv:2203.15556. Retrieved from https://arxiv.org/abs/2203.15556 · arXiv
  81. Chenhui Hu, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. 2024. WilKE: Wise-Layer Knowledge Editor for Lifelong Knowledge Editing. arXiv:2402.10987. Retrieved from https://arxiv.org/abs/2402.10987 · arXiv
  82. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv:2106.09685. Retrieved from https://arxiv.org/abs/ 2106.09685 · arXiv
  83. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=nZeVKeeFYf9
  84. Yebowen Hu, Tim Ganter, Hanieh Deilamsalehy, Franck Dernoncourt, Hassan Foroosh, and Fei Liu. 2023. Meeting- Bank: A Benchmark Dataset for Meeting Summarization. arXiv:2305.17529. Retrieved from https://arxiv.org/abs/ 2305.17529 · arXiv
  85. Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jin- song Su. 2024. Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal. arXiv:2403.01244. Retrieved from https://arxiv.org/abs/2403.01244 · arXiv
  86. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine Reading Comprehen- sion with Contextual Commonsense Reasoning. arXiv:1909.00277. Retrieved from https://arxiv.org/abs/1909.00277 · arXiv
  87. Quzhe Huang, Mingxu Tao, Zhenwei An, Chen Zhang, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. 2023. Lawyer LLaMA technical report. arXiv:2305.15062. Retrieved from https://arxiv.org/abs/2305.15062 · arXiv
  88. Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2023. Transformer-patcher: One mistake worth one neuron. arXiv:2301.09785. Retrieved from https://arxiv.org/abs/2301.09785 · arXiv
  89. Drew A. Hudson and Christopher D. Manning. 2019. GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. arXiv:1902.09506. Retrieved from https://arxiv.org/abs/1902.09506 · arXiv
  90. Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L. Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. 2024. Simple and scalable strategies to continually pre-train large language models. arXiv:2403.08763. Retrieved from https://arxiv.org/abs/2403.08763 · arXiv
  91. Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. 2022. TemporalWiki: A lifelong benchmark for training and evaluating ever-evolving language models. EMNLP 2022.
  92. Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2022. Towards continual knowledge learning of language models. In Proceedings of the ICLR.
  93. Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan O'Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo, and Wen Gao. 2024. AI Alignment: A Comprehensive Survey. arXiv:2310.19852. Retrieved from https://arxiv.org/abs/2310.19852 · arXiv
  94. Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Victoria Lin, Wen tau Yih, and Srinivasan Iyer. 2024. Instruction-tuned Language Models are Better Knowledge Learners. arXiv:2402.12847. Retrieved from https://arxiv.org/abs/2402.12847 · arXiv
  95. Xisen Jin and Xiang Ren. 2024. What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement. arXiv:2402.01865. Retrieved from https://arxiv.org/abs/2402.01865 · arXiv
  96. Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. 2022. Lifelong pretraining: Continually adapting language models to emerging corpora. In Proceedings of the BigScience Episode #5-Workshop on Challenges and Perspectives in Creating Large Language Models, Angela Fan, Suzana Ilic, Thomas Wolf, and Matthias Gallé (Eds.). Association for Computational Linguistics, virtual+Dublin, 1-16. DOI:https: //doi.org/10.18653/v1/2022.bigscience-1.1 · DOI
  97. Eric R. Kandel, James H. Schwartz, Thomas M. Jessell, Steven Siegelbaum, A. James Hudspeth, Sarah Mack, et al. 2000. Principles of Neural Science. McGraw-hill New York.
  98. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv:2001.08361. Retrieved from https://arxiv.org/abs/2001.08361 · arXiv
  99. Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. ReferItGame: Referring to objects in pho- tographs of natural scenes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Alessandro Moschitti, Bo Pang, and Walter Daelemans (Eds.). Association for Computational Linguistics, Doha, Qatar, 787-798. DOI:https://doi.org/10.3115/v1/D14-1086 · DOI
  100. Zixuan Ke, Haowei Lin, Yijia Shao, Hu Xu, Lei Shu, and Bing Liu. 2022. Continual training of language models for few-shot learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 10205-10216. DOI:https://doi.org/10.18653/v1/2022.emnlp-main.695 · DOI · Paper page
  101. Zixuan Ke and Bing Liu. 2023. Continual Learning of Natural Language Processing Tasks: A Survey. arXiv:2211.12701. Retrieved from https://arxiv.org/abs/2211.12701 · arXiv
  102. Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Shu Lei. 2021. Achieving forgetting prevention and knowledge transfer in continual learning. In Proceedings of the NeurIPS.
  103. Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2022. Continual pre-training of language models. In Proceedings of the 11th International Conference on Learning Representations.
  104. Tannon Kew, Marek Kostrzewa, and Sarah Ebling. 2023. 20 Minuten: A multi-task news summarisation dataset for german. In Proceedings of the 8th Edition of the Swiss Text Analytics Conference, Hatem Ghorbel, Maria Sokhn, Mark Cieliebak, Manuela Hürlimann, Emmanuel de Salis, and Jonathan Guerne (Eds.). Association for Computational Linguistics, Neuchatel, Switzerland, 1-13. Retrieved from https://aclanthology.org/2023.swisstext-1.1
  105. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 252-262.
  106. Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2017. Learning what is essential in questions. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), Roger Levy and Lucia Specia (Eds.). Association for Computational Linguistics, Vancouver, Canada, 80-89. DOI:https://doi.org/10.18653/ v1/K17-1010 · DOI
  107. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. QASC: A Dataset for Question Answering via Sentence Composition. arXiv:1910.11473. Retrieved from https://arxiv.org/abs/1910.11473 · arXiv
  108. Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, and Bing Liu. 2022. A theoretical study on solving con- tinual learning. In Proceedings of the Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.). Vol. 35, Curran Associates, Inc., 5065-5079. Retrieved from https: //proceedings.neurips.cc/paper_files/paper/2022/file/20f44da80080d76bbc35bca0027f14e6-Paper-Conference.pdf
  109. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114, 13 (2017), 3521-3526.
  110. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Ep- stein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics 7 (2019), 453-466.
  111. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv:1704.04683. Retrieved from https://arxiv.org/abs/1704.04683 · arXiv
  112. Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Tomas Kocisky, Sebastian Ruder, et al. 2021. Mind the gap: Assessing temporal gen- eralization in neural language models. Advances in Neural Information Processing Systems 34 (2021), 29348-29363.
  113. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. arXiv:1706.04115. Retrieved from https://arxiv.org/abs/1706.04115 · arXiv
  114. Chen-An Li and Hung-Yi Lee. 2024. Examining Forgetting in Continual Pre-training of Aligned Large Language Models. arXiv:2401.03129. Retrieved from https://arxiv.org/abs/2401.03129 · arXiv
  115. Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2022. Large language models with controllable working memory. arXiv:2211.05110. Retrieved from https://arxiv.org/ abs/2211.05110 · arXiv
  116. Haitao Li, Qingyao Ai, Jia Chen, Qian Dong, Zhijing Wu, Yiqun Liu, Chong Chen, and Qi Tian. 2024. BLADE: Enhancing black-box large language models with small domain-specific models. arXiv:2403.18365. Retrieved from https://arxiv.org/abs/2403.18365 · arXiv
  117. Jiangtong Li, Yuxuan Bian, Guoxuan Wang, Yang Lei, Dawei Cheng, Zhijun Ding, and Changjun Jiang. 2023. CFGPT: Chinese Financial Assistant with Large Language Model. arXiv:2309.10654. Retrieved from https://arxiv.org/abs/2309. 10654 · arXiv
  118. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv:2301.12597. Retrieved from https://arxiv.org/abs/ 2301.12597 · arXiv
  119. Linyang Li and Xipeng Qiu. 2023. Continual Model Evolvement with Inner-Product Restriction. Retrieved from https://openreview.net/forum?id=fn0BQK5T8p
  120. Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 12 (2017), 2935-2947.
  121. Bill Yuchen Lin, Sida Wang, Xi Lin, Robin Jia, Lin Xiao, Xiang Ren, and Scott Yih. 2022. On continual model refinement in out-of-distribution data streams. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, 3128-3139. DOI:https://doi.org/10.18653/v1/2022.acl-long.223 · DOI
  122. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Text Summariza- tion Branches Out. 74-81.
  123. Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. 2019. Reasoning Over Paragraph Effects in Situations. arXiv:1908.05852. Retrieved from https://arxiv.org/abs/1908.05852 · arXiv
  124. Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, and Tong Zhang. 2024. Mitigating the Alignment Tax of RLHF. arXiv:2309.06256. Retrieved from https://arxiv.org/abs/2309.06256 · arXiv
  125. Zhouhan Lin, Cheng Deng, Le Zhou, Tianhang Zhang, Yi Xu, Yutong Xu, Zhongmou He, Yuanyuan Shi, Beiya Dai, Yunchong Song, Boyi Zeng, Qiyuan Chen, Tao Shi, Tianyu Huang, Yiwei Xu, Shu Wang, Luoyi Fu, Weinan Zhang, Junxian He, Chao Ma, Yunqiang Zhu, Xinbing Wang, and Chenghu Zhou. 2023. GeoGalactica: A Scientific Large Language Model in Geoscience. arXiv:2401.00434. Retrieved from https://arxiv.org/abs/2401.00434 · arXiv
  126. Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al. 2024. Rho-1: Not all tokens are what you need. arXiv:2404.07965. Retrieved from https://arxiv.org/abs/ 2404.07965 · arXiv
  127. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. arXiv:2304.08485. Re- trieved from https://arxiv.org/abs/2304.08485 · arXiv
  128. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Infor- mation Processing Systems 35 (2022), 1950-1965.
  129. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692. Re- trieved from https://arxiv.org/abs/1907.11692 · arXiv
  130. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Dan
  131. Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds.). Association for Computational Linguistics, Online, 4969-4983. DOI:https://doi.org/10.18653/v1/2020.acl-main.447 · DOI · Paper page
  132. Vincenzo Lomonaco, Davide Maltoni, and Lorenzo Pellegrini. 2020. Rehearsal-Free Continual Learning over Small Non-I.I.D. Batches. arXiv:1907.03799. Retrieved from https://arxiv.org/abs/1907.03799 · arXiv
  133. David Lopez-Paz and Marc'Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. Advances in Neural Information Processing Systems 30 (2017).
  134. Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-collados. 2022. TimeLMs: Diachronic language models from Twitter. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Valerio Basile, Zornitsa Kozareva, and Sanja Stajner (Eds.). As- sociation for Computational Linguistics, Dublin, Ireland, 251-260. DOI:https://doi.org/10.18653/v1/2022.acl-demo.25 · DOI · Paper page
  135. Dakuan Lu, Hengkui Wu, Jiaqing Liang, Yipei Xu, Qianyu He, Yipeng Geng, Mengkun Han, Yingsi Xin, and Yanghua Xiao. 2023. BBT-Fin: Comprehensive construction of chinese financial domain pre-trained language model, corpus and benchmark. arXiv:2302.09432. Retrieved from https://arxiv.org/abs/2302.09432 · arXiv
  136. Pengyuan Lu, Michele Caprio, Eric Eaton, and Insup Lee. 2023. IBCL: Zero-shot Model Generation for Task Tradeoffs in Continual Learning. arXiv:2310.02995. Retrieved from https://arxiv.org/abs/2310.02995 · arXiv
  137. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. arXiv:2209.09513. Retrieved from https://arxiv.org/abs/2209.09513 · arXiv
  138. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. arXiv:2102.04664. Retrieved from https://arxiv.org/abs/ 2102.04664 · arXiv
  139. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv:2308.09583. Retrieved from https://arxiv.org/abs/2308.09583 · arXiv
  140. Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022. BioGPT: Generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics 23, 6 (2022). DOI:https: //doi.org/10.1093/bib/bbac409 · DOI · Paper page
  141. Yun Luo, Zhen Yang, Xuefeng Bai, Fandong Meng, Jie Zhou, and Yue Zhang. 2023. Investigating Forgetting in Pre- Trained Representations Through Continual Learning. arXiv:2305.05968. Retrieved from https://arxiv.org/abs/2305. 05968 · arXiv
  142. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning. arXiv:2308.08747. Retrieved from https://arxiv. org/abs/2308.08747 · arXiv
  143. Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. 2023. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. arXiv:2308.09442. Retrieved from https://arxiv.org/ abs/2308.09442 · arXiv
  144. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. WizardCoder: Empowering Code Large Language Models with Evol-Instruct. arXiv:2306.08568. Retrieved from https://arxiv.org/abs/2306.08568 · arXiv
  145. Shirong Ma, Shen Huang, Shulin Huang, Xiaobin Wang, Yangning Li, Hai-Tao Zheng, Pengjun Xie, Fei Huang, and Yong Jiang. 2023. EcomGPT-CT: Continual Pre-training of E-commerce Large Language Models with Semi-structured Data. arXiv:2312.15696. Retrieved from https://arxiv.org/abs/2312.15696 · arXiv
  146. Andrew Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 142-150.
  147. Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. 2022. Online continual learn- ing in image classification: An empirical survey. Neurocomputing 469 (2022), 28-51. DOI:https://doi.org/10.1016/j. neucom.2021.10.021 · DOI
  148. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. 2016. Generation and Comprehension of Unambiguous Object Descriptions. arXiv:1511.02283. Retrieved from https://arxiv.org/abs/ 1511.02283 · arXiv
  149. Vittorio Mazzia, Alessandro Pedrani, Andrea Caciolai, Kay Rottmann, and Davide Bernardi. 2023. A survey on knowl- edge editing of neural networks. arXiv:2310.19704. Retrieved from https://arxiv.org/abs/2310.19704 · arXiv
  150. David McCaffary. 2021. Towards continual task learning in artificial neural networks: Current approaches and in- sights from neuroscience. arXiv:2112.14146. Retrieved from https://arxiv.org/abs/2112.14146 · arXiv
  151. James L. McClelland, Bruce L. McNaughton, and Randall C. O'Reilly. 1995. Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review 102, 3 (1995), 419.
  152. Michael McCloskey and Neal J. Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, Vol. 24. Academic Press, 109-165. DOI:https://doi.org/10. 1016/S0079-7421(08)60536-8 · DOI · Paper page
  153. Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. 2023. An empirical investigation of the role of pre-training in lifelong learning. Journal of Machine Learning Research 24, 214 (2023), 1-50. Retrieved from http://jmlr.org/papers/v24/22-0496.html
  154. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems 35 (2022), 17359-17372.
  155. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2022. Mass-editing memory in a transformer. arXiv:1701.00133. Retrieved from https://arxiv.org/abs/1701.00133 · arXiv
  156. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? arXiv:2202.12837. Retrieved from https://arxiv.org/abs/2202.12837 · arXiv
  157. Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019. OCR-VQA: Visual question answering by reading text in images. In Proceedings of the ICDAR.
  158. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021. Natural Instructions: Benchmarking generalization to new tasks from natural language instructions. arXiv:2104.08773. Retrieved from https://arxiv.org/ abs/2104.08773 · arXiv
  159. Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. 2022. NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks. arXiv:2204.05660. Retrieved from https://arxiv.org/abs/2204.05660 · arXiv
  160. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2021. Fast model editing at scale. arXiv:2110.11309. Retrieved from https://arxiv.org/abs/2110.11309 · arXiv
  161. Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. 2022. Memory-based model editing at scale. In Proceedings of the International Conference on Machine Learning. PMLR, 15817-15831.
  162. Jisoo Mok, Jaeyoung Do, Sungjin Lee, Tara Taghavi, Seunghak Yu, and Sungroh Yoon. 2023. Large-scale lifelong learning of in-context instructions and how to tackle it. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 12573-12589. DOI:https://doi.org/10.18653/v1/2023.acl- long.703 · DOI
  163. Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Zhen Ming (Jack) Jiang. 2023. GitHub copilot AI pair programmer: Asset or liability? Journal of Systems and Software 203 (2023), 111734. DOI:https://doi.org/10.1016/j.jss.2023.111734 · DOI · Paper page
  164. Taishi Nakamura, Mayank Mishra, Simone Tedeschi, Yekun Chai, Jason T. Stillerman, Felix Friedrich, Prateek Yadav, Tanmay Laud, Vu Minh Chien, Terry Yue Zhuo, et al. 2024. Aurora-M: The first open source multilingual language model red-teamed according to the US executive order. arXiv:2404.00399. Retrieved from https://arxiv.org/abs/2404. 00399 · arXiv
  165. Tuan Dung Nguyen, Yuan-Sen Ting, Ioana Ciuca, Charlie O'Neill, Zechang Sun, Maja Jablonska, Sandor Kruk, Ernest Perkowski, Jack W. Miller, Jason Li, Josh Peek, Kartheik Iyer, Tomasz Rózanski, Pranav Khetarpal, Sharaf Zaman, David Brodrick, Sergio J. Rodríguez Méndez, Thang Bui, Alyssa Goodman, Alberto Accomazzi, Jill P. Naiman, Jesse Cranney, Kevin Schawinski, and UniverseTBD. 2023. AstroLLaMA: Towards specialized foundation models in astron- omy. arXiv:2309.06126. Retrieved from https://arxiv.org/abs/2309.06126 · arXiv
  166. Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process- ing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, Hong Kong, China, 188-197. DOI:https://doi.org/10.18653/v1/D19-1018 · DOI
  167. Zixuan Ni, Haizhou Shi, Siliang Tang, Longhui Wei, Qi Tian, and Yueting Zhuang. 2021. Revisiting catastrophic forgetting in class incremental learning. arXiv:2107.12308. Retrieved from https://arxiv.org/abs/2107.12308 · arXiv
  168. Zixuan Ni, Longhui Wei, Siliang Tang, Yueting Zhuang, and Qi Tian. 2023. Continual vision-language representa- tion learning with off-diagonal information. In Proceedings of the 40th International Conference on Machine Learning. 26129-26149.
  169. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An open large language model for code with multi-turn program synthesis. ICLR (2023).
  170. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. arXiv:2203.02155. Retrieved from https://arxiv.org/abs/2203.02155 · arXiv
  171. Christophe Pallier, Stanislas Dehaene, J-B Poline, Denis LeBihan, A-M Argenti, Emmanuel Dupoux, and Jacques Mehler. 2003. Brain imaging of language plasticity in adopted adults: Can a second language replace the first? Cerebral Cortex 13, 2 (2003), 155-161.
  172. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 311- 318.
  173. Indraneil Paul, Jun Luo, Goran Glavaš, and Iryna Gurevych. 2024. IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators. arXiv:2403.03894. Retrieved from https://arxiv.org/abs/2403. 03894 · arXiv
  174. Anastasia Pentina. 2016. Theoretical foundations of multi-task lifelong learning. Ph. D. Dissertation.
  175. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases?. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
  176. Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, Hong Kong, China, 2463-2473. DOI:https://doi.org/10.18653/v1/D19-1250 · DOI · Paper page
  177. Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet K. Dokania, Philip H. S. Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi. 2023. Computationally budgeted continual learning: What does matter?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3698-3707.
  178. Ameya Prabhu, Zhipeng Cai, Puneet Dokania, Philip Torr, Vladlen Koltun, and Ozan Sener. 2023. Online Continual Learning Without the Storage Constraint. arXiv:2305.09253. Retrieved from https://arxiv.org/abs/2305.09253 · arXiv
  179. Chengwei Qin and Shafiq Joty. 2021. LFPT5: A unified framework for lifelong few-shot language learning based on prompt tuning of T5. In Proceedings of the International Conference on Learning Representations.
  180. Yujia Qin, Cheng Qian, Xu Han, Yankai Lin, Huadong Wang, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2023. Recyclable tuning for continual pre-training. arXiv:2305.08702. Retrieved from https://arxiv.org/abs/2305.08702 · arXiv
  181. Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022. ELLE: Efficient lifelong pre-training for emerging data. In Findings of the Association for Computational Linguistics: ACL 2022, Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, 2789-2810. DOI:https://doi.org/10.18653/v1/2022.findings-acl.220 · DOI
  182. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervi- sion. In Proceedings of the International Conference on Machine Learning. PMLR, 8748-8763.
  183. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
  184. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36 (2024).
  185. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36 (2024).
  186. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21, 140 (2020), 1-67.
  187. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. arXiv:1806.03822. Retrieved from https://arxiv.org/abs/1806.03822 · arXiv
  188. Rahul Ramesh and Pratik Chaudhari. 2021. Model zoo: A growing "brain" that learns continually. arXiv:2106.03027. Retrieved from https://arxiv.org/abs/2106.03027 · arXiv
  189. Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, David Kreil, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. 2021. Hopfield Networks is All You Need. arXiv:2008.02217. Retrieved from https://arxiv.org/abs/2008.02217 · arXiv
  190. Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. 2017. icarl: Incremental clas- sifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2001-2010.
  191. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv:2403.05530. Retrieved from https://arxiv.org/abs/2403.05530 · arXiv
  192. Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. 2018. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv:1810.11910. Retrieved from https://arxiv.org/abs/1810.11910 · arXiv
  193. Hippolyt Ritter, Aleksandar Botev, and David Barber. 2018. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems 31 (2018).
  194. Subendhu Rongali, Abhyuday Jagannatha, Bhanu Pratap Singh Rawat, and Hong Yu. 2021. Continual Domain-Tuning for Pretrained Language Models. arXiv:2004.02288. Retrieved from https://arxiv.org/abs/2004.02288 · arXiv
  195. Guy D. Rosin, Ido Guy, and Kira Radinsky. 2022. Time masking for temporal language models. In Proceedings of the 15th ACM International Conference on Web Search and Data Mining (WSDM'22). Association for Computing Machin- ery, New York, NY, USA, 833-841. DOI:https://doi.org/10.1145/3488560.3498529 · DOI
  196. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cris- tian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2024. Code Llama: Open Foundation Models for Code. arXiv:2308.12950. Retrieved from https://arxiv.org/abs/2308.12950 · arXiv
  197. Andre Niyongabo Rubungo, Craig Arnold, Barry P. Rand, and Adji Bousso Dieng. 2023. LLM-Prop: Predicting phys- ical and electronic properties of crystalline solids from their text descriptions. arXiv:2310.14029. Retrieved from https://arxiv.org/abs/2310.14029 · arXiv
  198. Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2016. Progressive neural networks. arXiv:1606.04671. Retrieved from https://arxiv. org/abs/1606.04671 · arXiv
  199. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. WinoGrande: An Adversarial Wino- grad Schema Challenge at Scale. arXiv:1907.10641. Retrieved from https://arxiv.org/abs/1907.10641 · arXiv
  200. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multitask Prompted Training Enables Zero-Shot Task Generalization. arXiv:2110.08207. Retrieved from https://arxiv.org/abs/2110.08207 · arXiv

200 of 300 shown

Cited by

See all 135

Metadata

Authors
Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y., Wang, Z., Ebrahimi, S., & Wang, H.
Affiliations
  • Rutgers, The State University of New Jersey (US)
  • Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Volume
58
Issue
5
Pages
1-42
ISSN
0360-0300 (ISSN-L)
DOI
10.48550/arxiv.2404.16789
arXiv
2404.16789
OpenAlex
W4410371768
Cited by
135
References
0
Citations by year
2026: 69 · 2025: 54 · 2024: 12
FWCI
39.88
Type
journal article
Language
en
Open access
Licence
Locations
  • ACM Computing Surveys (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
continual learning, large language models, catastrophic forgetting, continual pre-training, domain-adaptive pretraining, continual fine-tuning, knowledge transfer, domain adaptation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Continual Learning of Large Language Models: A Comprehensive Survey. https://app.cochat.ai/papers/10.48550/arxiv.2404.16789
The paper
Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y., Wang, Z., Ebrahimi, S., & Wang, H. (2024). Continual Learning of Large Language Models: A Comprehensive Survey. arXiv. https://doi.org/10.48550/arxiv.2404.16789