CoChat
Sign Up
PreprintCited by 33Domain Adaptation and Few-Shot LearningTopic ModelingNatural Language Processing Techniques

Progressive Prompts: Continual Learning for Language Models

Open access
Razdaibiedina, Anastasia, Mao, Yuning, Hou, Rui, Khabsa, Madian, Lewis, Mike, Almahairi, Amjad·2023·arXiv
DOI 10.48550/arxiv.2301.12314arXiv 2301.12314
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce Progressive Prompts - a simple and efficient approach for continual learning in language models. Our method allows forward transfer and resists catastrophic forgetting, without relying on data replay or a large number of task-specific parameters. Progressive Prompts learns a new soft prompt for each task and sequentially concatenates it with the previously learned prompts, while keeping the base model frozen. Experiments on standard continual learning benchmarks show that our approach outperforms state-of-the-art methods, with an improvement >20% in average test accuracy over the previous best-preforming method on T5 model. We also explore a more challenging continual learning setup with longer sequences of tasks and show that Progressive Prompts significantly outperforms prior methods.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2301.12314

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Architecture-based approaches dynamically expand the model capacity or isolate existing model weights to reduce interference.

    “Architecture-based approaches (Wang et al., 2023e; Razdaibiedina et al., 2023) dynamically expand the model capacity or isolate existing model weights to reduce interference.”

    From Orthogonal Subspace Learning for Language Model Continual Learning (2023) · cites this paper as Razdaibiedina et al., 2023

  • Architecture-based approaches that train distinct expert models for different tasks restrict their generalization ability to unseen tasks.

    “However, such approaches essentially train distinct expert models for different tasks, which restricts their generalization ability to unseen tasks.”

    From Orthogonal Subspace Learning for Language Model Continual Learning (2023) · cites this paper as Razdaibiedina et al., 2023

Cites

Still being checked.

Cited by

Sorted by citations

Related

Authors
Razdaibiedina, A., Mao, Y., Hou, R., Khabsa, M., Lewis, M., & Almahairi, A.
Affiliations
University of Toronto (CA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2301.12314
arXiv
2301.12314
OpenAlex
W4318751794
Cited by
33
References
0
Citations by year
2025: 15 · 2024: 10 · 2023: 8
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
continual learning, language models, catastrophic forgetting, soft prompts, prompt learning, forward transfer, T5

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review