Parameter-efficient fine-tuning of large-scale pre-trained language models
Open accessPublished version of a preprint
Abstract
With the prevalence of pre-trained language models (PLMs) and the pre-training–fine-tuning paradigm, it has been continuously shown that larger models tend to yield better performance. However, as PLMs scale up, fine-tuning and storing all the parameters is prohibitively costly and eventually becomes practically infeasible. This necessitates a new branch of research focusing on the parameter-efficient adaptation of PLMs, which optimizes a small portion of the model parameters while keeping the rest fixed, drastically cutting down computation and storage costs. In general, it demonstrates that large-scale models could be effectively stimulated by the optimization of a few parameters. Despite the various designs, here we discuss and analyse the approaches under a more consistent and accessible term ‘delta-tuning’, where ‘delta’ a mathematical notation often used to denote changes, is borrowed to refer to the portion of parameters that are ‘changed’ during training. We formally describe the problem and propose a unified categorization criterion for existing delta-tuning methods to explore their correlations and differences. We also discuss the theoretical principles underlying the effectiveness of delta-tuning and interpret them from the perspectives of optimization and optimal control. Furthermore, we provide a holistic empirical study on over 100 natural language processing tasks and investigate various aspects of delta-tuning. With comprehensive study and analysis, our research demonstrates the theoretical and practical properties of delta-tuning in the adaptation of PLMs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some parameter-efficient fine-tuning techniques help reduce computational resource usage during incremental training while mitigating the loss of already stored parametric memory.
“On the other hand, some parameter-efficient fine-tuning [160, 161] techniques help reduce computational resource usage during incremental training while mitigating the loss of already stored parametric memory [131, 132].”
From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [161]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 123 · Cited by 1,191
Research works printed but not linked: 11
- Bengio, Y., Ducharme, R. & Vincent, P. A neural probabilistic language model. In Advances in Neural Information Processing Systems. 13 (2000).
- Vaswani, A. et al. Attention is all you need. In Advances in Neural Information Processing Systems. 30 (2017).
- Brown, T. et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems. 33, 1877-1901 (2020).
- Houlsby, N. et al. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning. (eds Chaudhuri, K. & Salakhutdinov, R.) 2790-2799 (2019).
- AAghajanyan, A., Gupt, S. & Zettlemoyer, L. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proc. the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 1, 7319-7328 (2021).
- Su, Y. et al. On transferability of prompt tuning for natural language understanding. In Proc. the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 3949-3969 (2022).
- Williams, A., Nangia, N. & Bowman, S. A broad-coverage challenge corpus for sentence understanding through inference. In Proc. the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 1, 1112-1122 (2018). Analysis https://doi.org/10.1038/s42256-023-00626-4
- Li, C., Farkhoor, H., Liu, R. & Yosinski, J. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations (2018).
- Yang, Z. & Liu, Y. On robust prefix-tuning for text classification. In International Conference on Learning Representations (2022).
- Boyd, S. P. & Barratt, C. H. Linear Controller Design: Limits of Performance Vol. 7 (Citeseer, 1991).
- He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T. & Neubig, G. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations (2022). https://doi.org/10.1038/s42256-023-00626-4 Extended Data Fig. 2 | The performance of T5 BASE with different delta-tuning methods at different training time. The performance of T5 BASE .with different delta- tuning methods (LR, AP, PF) and fine-tuning (FT) at different training time (seconds). https://doi.org/10.1038/s42256-023-00626-4 Extended Data Fig. 3 | Time consumption for fine-tuning (FT) and different delta-tuning methods. Time consumption for fine-tuning (FT) and different delta- tuning methods, including BitFit (BF), adapter (AP) and prompt-tuning (PT). We report the results with different input length. https://doi.org/10.1038/s42256-023-00626-4 Extended Data Fig. 4 | The performance of RoBERTa LARGE when sequentially applying different delta-tuning methods. The performance of RoBERTa LARGE when different delta-tuning methods (adapter (AP), BitFit (BF) and prompt-tuning (PT)) are applied sequentially. The experiments are conducted on SST-2. · DOI
- Authors
- Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., Yi, J., Zhao, W., Wang, X., Liu, Z., Zheng, H.-T., Chen, J., Liu, Y., Tang, J., Li, J., & Sun, M.
- Affiliations
- Beijing Academy of Artificial Intelligence (CN)
- Tsinghua University (CN)
- Tsinghua Shenzhen International Graduate School (CN)
- ORCID
- Venue
- Nature Machine Intelligence
- Published
- 2023-03-02
- Volume
- 5
- Issue
- 3
- Pages
- 220-235
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W4322766882
- Cited by
- 1,191
- References
- 123
- Citations by year
- 2026: 358 · 2025: 464 · 2024: 275 · 2023: 84 · 2022: 10
- FWCI
- 111.19
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Machine Intelligence (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- Funders
- National Natural Science Foundation of China (62276154)
- Tsinghua University
- National Key Research and Development Program of China (2020AAA0106500)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- parameter-efficient fine-tuning, pre-trained language models, large language models, model compression, optimization theory, optimal control, natural language processing tasks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
