CoChat
Sign Up
PreprintCited by 20Domain Adaptation and Few-Shot LearningTopic ModelingAdvanced Neural Network Applications

Parameter-Efficient Transfer Learning with Diff Pruning

Open access
Guo, Demi, Rush, Alexander M., Kim, Yoon·2020·arXiv
DOI 10.48550/arxiv.2012.07463arXiv 2012.07463
Open this paper in CoChat — free accountOpen PDF

Abstract

While task-specific finetuning of pretrained networks has led to significant empirical advances in NLP, the large size of networks makes finetuning difficult to deploy in multi-task, memory-constrained settings. We propose diff pruning as a simple approach to enable parameter-efficient transfer learning within the pretrain-finetune framework. This approach views finetuning as learning a task-specific diff vector that is applied on top of the pretrained parameter vector, which remains fixed and is shared across different tasks. The diff vector is adaptively pruned during training with a differentiable approximation to the L0-norm penalty to encourage sparsity. Diff pruning becomes parameter-efficient as the number of tasks increases, as it requires storing only the nonzero positions and weights of the diff vector for each task, while the cost of storing the shared pretrained model remains constant. It further does not require access to all tasks during training, which makes it attractive in settings where tasks arrive in stream or the set of tasks is unknown. We find that models finetuned with diff pruning can match the performance of fully finetuned baselines on the GLUE benchmark while only modifying 0.5% of the pretrained model's parameters per task.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2012.07463

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Diff pruning needs to update every entry of ∆ with their gradients and then prune them, resulting in similar computational cost as full fine-tuning.

    “Second, it needs to update every entry of ∆ with their gradients and then prune them.This results in similar computational cost as full fine-tuning (Guo et al., 2020).”

    From AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (2023) · cites this paper as Guo et al., 2020

    Section 3.1 (p. 3) describes gradient-based training with a dense weight vector, and Section 3.2 (p. 3) describes magnitude pruning after training. The computational-cost comparison needs qualification: Section 6.7 (p. 7) reports that diff pruning was approximately 1.5–2 times slower per minibatch than regular fine-tuning, not simply equivalent in cost. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Diff pruning can increase parameter efficiency substantially by adaptively retaining important updates and pruning unimportant ones.

    “As such, diff pruning can increase the parameter efficiency substantially by adaptively retaining important updates and pruning unimportant ones.”

    From AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (2023) · cites this paper as Guo et al., 2020

  • Diff pruning relies on low-level implementation to speed up computation of unstructured sparse matrices, which is not well supported by existing deep learning frameworks.

    “First, it relies on low-level implementation to speed up the computation of unstructured sparse matrices, which is not well supported by existing deep learning frameworks.”

    From AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (2023) · cites this paper as Guo et al., 2020

    Section 6.7 (p. 7) reports that diff pruning trained 1.5–2 times more slowly per minibatch than regular finetuning, but does not discuss speeding up sparse-matrix computation. The passages on structured and non-structured pruning (§6.2, p. 6) also do not say that diff pruning relies on low-level implementations or that deep learning frameworks lack support for them; searches for those terms found no such passage. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 65

  1. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A Simple but Tough-to-Beat Baseline for Sentence Embeddings . In Proceedings of ICLR , 2017.
  2. Steven Cao, Victor Sanh, and Alexander M. Rush. Low-Complexity Probing via Finding Subnetworks. In Proceedings of NAACL , 2021.
  3. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. Universal sentence encoder for English. In Proceedings of EMNLP: System Demonstrations , 2018.
  4. Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The Lottery Ticket Hypothesis for Pretrained BERT Networks. arXiv:2007.12223 , 2020. · arXiv
  5. Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le. BAM! Born-Again Multi-Task Networks for Natural Language Understanding. In Proceedings of ACL , 2019.
  6. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In Proceedings of EMNLP , 2017.
  7. Andrew Dai and Quoc V. Le. Semi-Supervised Sequence Learning. In Proceedings of NIPS , 2015.
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL , 2019.
  9. Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Deming Chen, Marianne Winslett, Hassan Sajjad, and Preslav Nakov. Compressing Large-Scale Transformer-Based Models: A Case Study on BERT. arXiv:2002.11985 , 2020. · arXiv
  10. Mitchell A. Gordon, Kevin Duh, and Nicholas Andrews. Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning. In Proceedings of Rep4NLP 2020 Workshop at ACL 2020 , 2020.
  11. Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. SpotTune: Transfer Learning Through Adaptive FineTuning. In Proceedings of CVPR , 2019.
  12. Song Han, Huizi Mao, and William J. Dally. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. In Proceedings of ICLR , 2016.
  13. Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. In Proceedings of ACL , 2016.
  14. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, and Mona Attariyanand Sylvain Gelly. Parameter-efficient transfer learning for nlp. In Proceedings of ICML , 2019.
  15. Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification. In Proceedings of ACL , 2018.
  16. Eric Jang, Shixiang Gu, and Ben Poole. Categorical Reparameterization with Gumbel-Softmax. In Proceedings of ICLR , 2017.
  17. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for Natural Language Understanding. In Proceedings of EMNLP (Findings) , 2020.
  18. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences , 14:3521-3526, 2017.
  19. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Skip-Thought Vectors. In Proceedings of NeurIPS , 2015.
  20. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In Proceedings of ICLR , 2020.
  21. Quoc V. Le and Tomas Mikolov. Distributed Representations of Sentences and Documents. In Proceedings of ICML , 2014.
  22. Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective Regularization to Finetune Largescale Pretrained Language Models. In Proceedings of ICLR , 2020.
  23. Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In Advances in Neural Information Processing Systems . 2017.
  24. Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv:2101.00190 , 2021. · arXiv
  25. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic Knowledge and Transferability of Contextual Representations. In Proceedings of ACL , 2019a.
  26. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-Task Deep Neural Networks for Natural Language Understanding. In Proceedings of ACL , 2019b.
  27. A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 , 2019c. · arXiv
  28. David Lopez-Paz and Marc'Aurelio Ranzato. Gradient Episodic Memory for Continual Learning. In Proceedings of NeurIPS , 2017.
  29. Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. In Proceedings of ICLR , 2017.
  30. Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights. In Proceedings of ECCV , 2018.
  31. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Proceedings of NeurIPS . 2017.
  32. Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, and Rico Sennrich. Regularization techniques for fine-tuning in neural machine translation. In Proceedings of EMNLP , 2017.
  33. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781 , 2013. · arXiv
  34. Jonas Pfeiffer, Andreas Ruckle, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, and Iryna Gurevych Kyunghyun Cho. AdapterHub: A Framework for Adapting Transformers. arXiv:2007.07779 , 2020b. · arXiv
  35. Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer. arXiv:2005.00052 , 2020c. · arXiv
  36. Guanghui Qin and Jason Eisner. Learning How to Ask: Querying LMs with Mixtures of Soft Prompts. In Proceedings of NAACL , 2021.
  37. Evani Radiya-Dixit and Xin Wang. How fine can finetuning be? Learning efficient language models. In Proceedings of AISTATS , 2020.
  38. Colin Raffel, Noam Shazeer, Katherine Lee Adam Roberts, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research , 21, 2020.
  39. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. arXiv:1910.02054 , 2019. · arXiv
  40. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP , 2016.
  41. S. Rebuffi, A. Vedaldi, and H. Bilen. Efficient Parametrization of Multi-domain Deep Neural Networks. In Proceedings of CVPR , 2018.
  42. Nils Reimers and Iryna Gurevych. SentenceBERT: Sentence Embeddings using Siamese BERTNetworks. In Proceedings of EMNLP , 2019.
  43. Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive Neural Networks. arXiv:1606.04671 , 2016. · arXiv
  44. Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. Poor Man's BERT: Smaller and Faster Transformer Models. arXiv:2004.03844 , 2020. · arXiv
  45. Timo Schick and Hinrich Schutze. It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. arXiv:2009.07118 , 2020. · arXiv
  46. Jonathan Schwarz, Jelena Luketina, Wojciech M. Czarnecki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & Compress: A scalable framework for continual learning. In Proceedings of ICML , 2018.
  47. Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual Learning with Deep Generative Replay. In Proceedings of NeurIPS . 2017.
  48. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053 , 2019. · arXiv
  49. Asa Cooper Stickland and Iain Murray. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. In Proceedings of ICML , 2019.
  50. Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J. Pal. Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning. In Proceedings of ICLR , 2018.
  51. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient Knowledge Distillation for BERT Model Compression. In Proceedings of EMNLP , 2019.
  52. Ximeng Sun, Rameswar Panda, and Rogerio Feris. AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning. In Proceedings of NeurIPS , 2020a.
  53. Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of ACL , 2019.
  54. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-Read Students Learn Better: On the Importance of Pre-training Compact Models. arXiv:1908.08962 , 2019. · arXiv
  55. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of ICLR , 2019a.
  56. Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured Pruning of Large Language Models. arXiv:1910.04732 , 2019b. · arXiv
  57. Georg Wiese, Dirk Weissenborn, and Mariana Neves. Neural domain adaptation for biomedical question answering. In Proceedings of CoNLL , August 2017.
  58. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. Towards Universal Paraphrastic Sentence Embeddings. In Proceedings of ICLR , 2016.
  59. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R´ emi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-art natural language processing. ArXiv , abs/1910.03771, 2019. · arXiv
  60. John M. Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Similarity Analysis of Contextual Word Representation Models. In Proceedings of ACL , 2020.
  61. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Proceedings of NeurIPS , 2019.
  62. Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks. In Proceedings of ECCV , 2020a.
  63. Minghua Zhang, Yunfang Wu, Weikang Li, and Wei Li. Learning universal sentence representations with mean-max attention autoencoder. In Proceedings of EMNLP , 2018.
  64. Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. An Unsupervised Sentence Embedding Method byMutual Information Maximization. In Proceedings of EMNLP , 2020b.
  65. Mengjie Zhao, Tao Lin, Martin Jaggi, and Hinrich Schutze. Masking as an Efficient Alternative to Finetuning for Pretrained Language Models. arXiv:2004.12406 , 2020. · arXiv

Cited by

Sorted by citations

Authors
Guo, D., Rush, A. M., & Kim, Y.
Affiliations
  • Harvard University (US)
  • Cornell University (US)
  • IBM (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2012.07463
arXiv
2012.07463
OpenAlex
W3113057009
Cited by
20
References
6
Citations by year
2025: 3 · 2024: 2 · 2023: 2 · 2022: 3 · 2021: 10
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (1704834, 2037519)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
parameter-efficient transfer learning, l0-norm sparsity, multi-task learning, memory-constrained deployment, pre-trained language models, GLUE benchmark

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review