Parameter-Efficient Transfer Learning with Diff Pruning
Open accessAbstract
Demi Guo, Alexander Rush, Yoon Kim. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
These approaches cannot provide the inference-time adaptation needed for agents operating in real-time, often on resource-constrained hardware or without access to training infrastructure.
“These approaches cannot provide the inference-time adaptation needed for agents operating in real-time, often on resource-constrained hardware or without access to training infrastructure (Guo et al., 2021; Dettmers et al., 2023; Liu et al., 2024; Frantar & Alistarh, 2023).”
From Continual Learning, Not Training: Online Adaptation For Agents (2025) · cites this paper as Guo et al., 2021
Section 3 describes diff pruning as learning a task-specific vector through empirical risk minimization, and Section 6.7 says it is slower to train than regular finetuning. This supports a distinction between training-based adaptation and inference-time adaptation, but the paper does not assess real-time agents or operation without training infrastructure. Its Abstract also describes diff pruning as attractive for on-device deployment where tasks arrive in a stream, so the resource-constrained setting is overstated as a categorical limitation. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Distributed Representations of Sentences and DocumentsPreprint
Le, Q. V., & Mikolov, T. (2014) · International Conference on Machine Learning · doi:10.48550/arxiv.1405.4053 · Cited by 10,307 · see citing works
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899 · see citing works
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · see citing works
Caruana, R. (1997) · Machine Learning · doi:10.1023/a:1007379606734 · Cited by 6,636 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · see citing works
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · see citing works
Categorical Reparameterization with Gumbel-Softmax
Jang, E., Gu, S., & Poole, B. (2016) · arXiv · doi:10.48550/arxiv.1611.01144 · Cited by 3,616 · see citing works
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · see citing works
Self-citationPrefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · see citing works
Catastrophic forgetting in connectionist networks
French, R. (1999) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(99)01294-2 · Cited by 2,311 · see citing works
Parameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270 · see citing works
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data
Conneau, A., Kiela, D., Schwenk, H., Barrault, L., & Bordes, A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1070 · Cited by 2,049 · see citing works
TinyBERT: Distilling BERT for natural language understanding
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.372 · Cited by 1,786 · see citing works
The Concrete Distribution: A Continuous Relaxation of Discrete Random VariablesPreprint
Maddison, C. J., Mnih, A., & Teh, Y. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.00712 · Cited by 1,771 · see citing works
(2018) · International Conference on Learning Representations · Cited by 1,621 · see citing works
Gradient Episodic Memory for Continual Learning
Lopez-Paz, D., & Ranzato, M. (2017) · arXiv · doi:10.48550/arxiv.1706.08840 · Cited by 1,403 · see citing works
Continual Learning with Deep Generative Replay
Shin, H., Lee, J. K., Kim, J., & Kim, J. (2017) · arXiv · doi:10.48550/arxiv.1705.08690 · Cited by 1,385 · see citing works
Universal Sentence Encoder for English
Cer, D. M., Yang, Y., Kong, S.-Y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Céspedes, M., Yuan, S., Tar, C., et al. (2018) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/d18-2029 · Cited by 1,344 · see citing works
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307 · see citing works
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1606.04671 · Cited by 1,307 · see citing works
Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S. (2015) · arXiv · doi:10.48550/arxiv.1506.06726 · Cited by 1,293 · see citing works
Multi-Task Deep Neural Networks for Natural Language Understanding
Liu, X., He, P., Chen, W., & Gao, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1441 · Cited by 1,216 · see citing works
(2017) · International Conference on Learning Representations · Cited by 1,075 · see citing works
Semi-supervised Sequence Learning
Dai, A. M., & Le, Q. V. (2015) · arXiv · doi:10.48550/arxiv.1511.01432 · Cited by 994 · see citing works
(2020) · doi:10.1109/sc41405.2020.00024 · Cited by 928 · see citing works
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · see citing works
Patient Knowledge Distillation for BERT Model Compression
Sun, S., Cheng, Y., Gan, Z., & Liu, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1441 · Cited by 758 · see citing works
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., & Zhou, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.195 · Cited by 757 · see citing works
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.185 · Cited by 747 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · arXiv · doi:10.48550/arxiv.1606.05250 · Cited by 678 · see citing works
Linguistic Knowledge and Transferability of Contextual Representations
Liu, N. F., Gardner, M., Belinkov, Y., Peters, M. E., & Smith, N. A. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1112 · Cited by 651 · see citing works
Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights
Mallya, A., Davis, D., & Lazebnik, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01225-0_5 · Cited by 644 · see citing works
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., & Gurevych, I. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.39 · Cited by 636 · see citing works
Learning Sparse Neural Networks through $L_0$ Regularization
Louizos, C., Welling, M., & Kingma, D. P. (2017) · arXiv · doi:10.48550/arxiv.1712.01312 · Cited by 628 · see citing works
Well-Read Students Learn Better: On the Importance of Pre-training Compact ModelsPreprint
Turc, I., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.08962 · Cited by 620 · see citing works
Learned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572 · cited 1× · see citing works
“Pretraining objectives include context prediction (Mikolov et al., 2013), autoencoding (Dai & Le, 2015), machine translation (McCann et al., 2017), and more recently, variants of language modeling (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019) objectives.”
BackgroundLearning Distributed Representations of Sentences from Unlabelled Data
Hill, F., Cho, K., & Korhonen, A. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1162 · Cited by 566 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018) · arXiv · doi:10.48550/arxiv.1804.07461 · Cited by 562 · see citing works
AdapterHub: A Framework for Adapting Transformers
Pfeiffer, J., Rücklé, A., Poth, C., Kamath, A., Vulić, I., Ruder, S., Cho, K., & Gurevych, I. (2020) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2020.emnlp-demos.7 · Cited by 506 · see citing works
SpotTune: Transfer Learning Through Adaptive Fine-Tuning
Guo, Y., Shi, H., Kumar, A., Grauman, K., Rosing, T., & Feris, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00494 · Cited by 506 · see citing works
Overcoming Catastrophic Forgetting by Incremental Moment MatchingPreprint
Lee, S.-W., Kim, J.-H., Jun, J., Ha, J.-W., & Zhang, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.08475 · Cited by 472 · see citing works
MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
Pfeiffer, J., Vulić, I., Gurevych, I., & Ruder, S. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.617 · Cited by 467 · see citing works
Progress & Compress: A scalable framework for continual learningPreprint
Schwarz, J., Luketina, J., Czarnecki, W. M., Grabska‐Barwińska, A., Teh, Y. W., Pascanu, R., & Hadsell, R. T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.06370 · Cited by 422 · see citing works
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
Qin, G., & Eisner, J. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.410 · Cited by 342 · see citing works
Towards Universal Paraphrastic Sentence EmbeddingsPreprint
Wieting, J., Bansal, M., Gimpel, K., & Livescu, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.08198 · Cited by 324 · see citing works
Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning
Gordon, M., Duh, K., & Andrews, N. (2020) · Proceedings of the 5th Workshop on Representation Learning for NLP · doi:10.18653/v1/2020.repl4nlp-1.18 · Cited by 303 · see citing works
BAM! Born-Again Multi-Task Networks for Natural Language Understanding
Clark, K., Luong, M.-T., Khandelwal, U., Manning, C. D., & Le, Q. V. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1595 · Cited by 219 · see citing works
Movement pruning: Adaptive sparsity by fine-tuning
Sanh, V., Wolf, T., & Rush, A. M. (2020) · arXiv · doi:10.48550/arxiv.2005.07683 · Cited by 218 · see citing works
Self-citationLearning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
Subramanian, S., Trischler, A., Bengio, Y., & Pal, C. J. (2018) · arXiv · doi:10.48550/arxiv.1804.00079 · Cited by 207 · see citing works
The Lottery Ticket Hypothesis for Pre-trained BERT NetworksPreprint
Chen, T., Frankle, J., Chang, S., Liu, S., Zhang, Y., Wang, Z., & Carbin, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.12223 · Cited by 198 · see citing works
Structured Pruning of Large Language Models
Wang, Z., Wohlwend, J., & Lei, T. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.496 · Cited by 194 · see citing works
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
Ganesh, P., Chen, Y., Lou, X., Khan, M. A., Yang, Y., Sajjad, H., Nakov, P., Chen, D., & Winslett, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00413 · Cited by 190 · see citing works
Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks
Zhang, J. O., (Sasha), A. S., Zamir, A., Guibas, L. J., & Malik, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58580-8_41 · Cited by 190 · see citing works
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
Sun, X., Panda, R., Feris, R., & Saenko, K. (2019) · arXiv · doi:10.48550/arxiv.1911.12423 · Cited by 169 · see citing works
An Unsupervised Sentence Embedding Method by Mutual Information Maximization
Zhang, Y., He, R., Liu, Z., Lim, K. H., & Bing, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.124 · Cited by 152 · see citing works
Mixout: Effective Regularization to Finetune Large-scale Pretrained Language ModelsPreprint
Lee, C., Cho, K., & Kang, W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11299 · Cited by 150 · see citing works
Neural Domain Adaptation for Biomedical Question Answering
Wiese, G. K., Weissenborn, D., & Neves, M. (2017) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k17-1029 · Cited by 112 · see citing works
BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task LearningPreprint
Stickland, A. C., & Murray, I. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.02671 · Cited by 109 · see citing works
Regularization techniques for fine-tuning in neural machine translation
Barone, A. V. M., Haddow, B., Germann, U., & Sennrich, R. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1156 · Cited by 98 · see citing works
BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
Stickland, A. C., & Murray, I. (2019) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 97 · see citing works
Poor Man's BERT: Smaller and Faster Transformer Models.Preprint
Sajjad, H., Dalvi, F., Durrani, N., & Nakov, P. (2020) · arXiv (Cornell University) · Cited by 77 · see citing works
Efficient Parametrization of Multi-domain Deep Neural Networks
Rebuffi, S.-A., Vedaldi, A., & Bilen, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00847 · Cited by 75 · see citing works
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models
Zhao, M., Lin, T., Mi, F., Jaggi, M., & Schütze, H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.174 · Cited by 67 · see citing works
Similarity Analysis of Contextual Word Representation Models
Wu, J., Belinkov, Y., Sajjad, H., Durrani, N., Dalvi, F., & Glass, J. R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.422 · Cited by 52 · see citing works
How fine can fine-tuning be? Learning efficient language modelsPreprint
Radiya-Dixit, E., & Wang, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.14129 · Cited by 31 · see citing works
Learning Universal Sentence Representations with Mean-Max Attention Autoencoder
Zhang, M., Wu, Y., Li, W., & Li, W. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1481 · Cited by 29 · see citing works
Masking as an Efficient Alternative to Finetuning for Pretrained Language ModelsPreprint
Zhao, M., Lin, T., Mi, F., Jaggi, M., & Schütze, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12406 · Cited by 19 · see citing works
Low-Complexity Probing via Finding Subnetworks
Cao, S., Sanh, V., & Rush, A. M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.74 · Cited by 18 · see citing works
Self-citationAn Unsupervised Sentence Embedding Method byMutual Information MaximizationPreprint
Zhang, Y., He, R., Liu, Z., Lim, K. H., & Bing, L. (2020) · arXiv (Cornell University) · Cited by 5 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
8 of the 89 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Andrew Dai and Quoc V. Le. Semi-Supervised Se- quence Learning. In Proceedings of NIPS, 2015.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep Contextualized Word Represen- tations. In Proceedings of NAACL, 2018.
- Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory Optimiza- tions Toward Training Trillion Parameter Models. arXiv:1910.02054, 2019. · arXiv
- Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Pa- tient Knowledge Distillation for BERT Model Com- pression. In Proceedings of EMNLP, 2019.
- Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. An Unsupervised Sentence Em- bedding Method byMutual Information Maximiza- tion. In Proceedings of EMNLP, 2020b.
Cited by
Sorted by citations
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., & Lim, S.-N. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19827-4_41 · Cited by 1,965 · see citing works
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387 · see citing works
Parameter-efficient fine-tuning of large-scale pre-trained language models
Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00626-4 · Cited by 1,191 · see citing works
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Ben Zaken, E., Goldberg, Y., & Ravfogel, S. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.1 · Cited by 765 · see citing works
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., & Gurevych, I. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.39 · Cited by 636 · see citing works
Towards a Unified View of Parameter-Efficient Transfer LearningPreprint
He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., & Neubig, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04366 · Cited by 514 · see citing works
Self-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372 · see citing works
“[339] are lightweight modules inserted into SSL models, and in downstream tasks, the parameters of SSL models are frozen, and only the adapters are trained.”
BackgroundVL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
Sung, Y.-L., Cho, J., & Bansal, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00516 · Cited by 329 · see citing works
Xu, L., Xie, H., Qin, S. J., Tao, X., & Wang, F. L. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3657354 · Cited by 318 · see citing works
Frozen CLIP Models are Efficient Video Learners
Lin, Z., Geng, S., Zhang, R., Gao, P., de Melo, G., Wang, X., Dai, J., Qiao, Y., & Li, H. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19833-5_23 · Cited by 213 · see citing works
Parameter-efficient fine-tuning in large language models: a survey of methodologies
Wang, L., Chen, S., Jiang, L., Pan, S., Cai, R., Yang, S., & Yang, F. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11236-4 · Cited by 186 · see citing works
On the Effectiveness of Parameter-Efficient Fine-Tuning
Fu, Z., Yang, H., So, A. M., Lam, W. P., Bing, L., & Collier, N. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26505 · Cited by 165 · see citing works
3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation
Gong, S., Zhong, Y., Ma, W., Li, J., Wang, Z., Zhang, J., Heng, P.-A., & Dou, Q. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103324 · Cited by 156 · see citing works
Block Pruning For Faster Transformers
Lagunas, F., Charlaix, E., Sanh, V., & Rush, A. M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.829 · Cited by 144 · see citing works
Self-citationRaise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning
Xu, R., Luo, F., Zhang, Z., Tan, C., Chang, B., Huang, S., & Huang, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.749 · Cited by 137 · see citing works
Efficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 124 · see citing works
Vision Transformers are Parameter-Efficient Audio-Visual Learners
Lin, Y.-B., Sung, Y.-L., Lei, J., Bansal, M., & Bertasius, G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00228 · Cited by 113 · see citing works
Survey of Different Large Language Model Architectures: Trends, Benchmarks, and Challenges
Shao, M., Basit, A., Karri, R., & Shafique, M. (2024) · IEEE Access · doi:10.1109/access.2024.3482107 · Cited by 110 · see citing works
Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning
Zhang, Z.-R., Tan, C., Xu, H., Wang, C., Huang, J. G., & Huang, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.107 · Cited by 102 · see citing works
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning
Mao, Y., Mathias, L., Hou, R. L., Almahairi, A., Ma, H., Han, J., Yih, S., & Khabsa, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.433 · Cited by 95 · see citing works
Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentation
Xu, Z., Chen, Z., Zhang, Y., Song, Y., Wan, X., & Li, G. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01605 · Cited by 91 · see citing works
E 2 VPT: An Effective and Efficient Approach for Visual Prompt Tuning
Han, C., Wang, Q., Cui, Y., Cao, Z., Wang, W., Qi, S., & Liu, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01604 · Cited by 87 · see citing works
A Survey on Green Deep LearningPreprint
Xu, J., Zhou, W., Fu, Z., Zhou, H., & Li, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05193 · Cited by 84 · see citing works
Sam-based instance segmentation models for the automation of structural damage detection
Ye, Z., Lovell, L., Faramarzi, A., & Ninić, J. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.102826 · Cited by 82 · see citing works
Sparse Low-rank Adaptation of Pre-trained Language Models
Ding, N., Lv, X., Wang, Q., Chen, Y., Zhou, B., Liu, Z., & Sun, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.252 · Cited by 81 · see citing works
Composable Sparse Fine-Tuning for Cross-Lingual Transfer
Ansell, A., Ponti, E. M., Korhonen, A., & Vulić, I. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.125 · Cited by 78 · see citing works
Training Neural Networks with Fixed Sparse MasksPreprint
Sung, Y.-L., Nair, V., & Raffel, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.09839 · Cited by 74 · see citing works
StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation
Maharana, A., Hannan, D., & Bansal, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_5 · Cited by 72 · see citing works
A Survey on Model Compression and Acceleration for Pretrained Language Models
Xu, C., & McAuley, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26255 · Cited by 70 · see citing works
Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning
He, H., Cai, J., Zhang, J., Tao, D., & Zhuang, B. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01086 · Cited by 69 · see citing works
Diversity-Aware Meta Visual Prompting
Huang, Q., Dong, X., Chen, D., Zhang, W., Wang, F., Hua, G., & Yu, N. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01047 · Cited by 67 · see citing works
Generative AI model privacy: a survey
Liu, Y., Huang, J., Li, Y., Wang, D., & Xiao, B. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-11024-6 · Cited by 65 · see citing works
Zhang, J., Wu, S., Gao, L., Shen, H. T., & Song, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01228 · Cited by 63 · see citing works
Revisiting Parameter-Efficient Tuning: Are We Really There Yet?
Chen, G., Liu, F., Meng, Z., & Liang, S. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.168 · Cited by 60 · see citing works
The Efficiency MisnomerPreprint
Dehghani, M., Arnab, A., Beyer, L., Vaswani, A., & Tay, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12894 · Cited by 58 · see citing works
Dual-Path Adaptation from Image to Video Transformers
Park, J., Lee, J., & Sohn, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00219 · Cited by 56 · see citing works
An Overview of Language Models: Recent Developments and Outlook
Wei, C., Wang, Y.-C., Wang, B., & Kuo, C. ‐. J. (2024) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000010 · Cited by 55 · see citing works
Petals: Collaborative Inference and Fine-tuning of Large Models
Borzunov, A., Baranchuk, D., Dettmers, T., Riabinin, M., Belkada, Y., Chumachenko, A., Samygin, P., & Raffel, C. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) · doi:10.18653/v1/2023.acl-demo.54 · Cited by 52 · see citing works
Parameter-Efficient Transfer Learning for Medical Visual Question Answering
Liu, J., Hu, T., Zhang, Y., Yang, F., Hao, J., Lv, J., & Liu, Z. (2023) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2023.3311333 · Cited by 51 · see citing works
Parameter-Efficient Fine-Tuning without Introducing New Latency
Liao, B., Meng, Y., & Monz, C. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.233 · Cited by 50 · see citing works
Żarski, M., Wójcik, B., Książek, K., & Miszczak, J. A. (2021) · Computer-Aided Civil and Infrastructure Engineering · doi:10.1111/mice.12755 · Cited by 43 · see citing works
BERT Busters: Outlier Dimensions that Disrupt Transformers
Kovaleva, O., Kulshreshtha, S., Rogers, A., & Rumshisky, A. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.300 · Cited by 41 · see citing works
AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks
Fu, C.-L., Chen, Z.-C., Lee, Y.-R., & Lee, H.-Y. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.199 · Cited by 40 · see citing works
Two-stage fine-tuning with ChatGPT data augmentation for learning class-imbalanced data
ValizadehAslani, T., Shi, Y., Wang, J., Ren, P., Zhang, Y., Hu, M., Zhao, L., & Liang, H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127801 · Cited by 39 · see citing works
Wang, T., Zhou, W., Zeng, Y., & Zhang, X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.873 · Cited by 38 · see citing works
A Comprehensive Overview and Analysis of Large Language Models: Trends and Challenges
Mohammed, A., & Kora, R. (2025) · IEEE Access · doi:10.1109/access.2025.3573955 · Cited by 38 · see citing works
CroApp: A CNN-Based Resource Optimization Approach in Edge Computing Environment
Jia, Y., Liu, B., Dou, W., Xu, X., Zhou, X., Qi, L., & Yan, Z. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3154473 · Cited by 36 · see citing works
Distributed Inference and Fine-tuning of Large Language Models Over The Internet
Borzunov, A., Ryabinin, M., Chumachenko, A., Baranchuk, D., Dettmers, T., Belkada, Y., Samygin, P., & Raffel, C. (2023) · arXiv · doi:10.48550/arxiv.2312.08361 · Cited by 35 · see citing works
AutoPEFT : Automatic Configuration Search for Parameter-Efficient Fine-Tuning
Zhou, H., Wan, X., Vulić, I., & Korhonen, A. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00662 · Cited by 35 · see citing works
LeSAM: Adapt Segment Anything Model for Medical Lesion Segmentation
Gu, Y., Wu, Q., Tang, H. N., Mai, X., Shu, H., Li, B., & Chen, Y. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3406871 · Cited by 35 · see citing works
MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval
Jin, X., Zhang, B., Gong, W., Xu, K., Deng, X., Wang, P., Zhang, Z., Shen, X., & Feng, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02563 · Cited by 35 · see citing works
HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizability
Chen, J., Shen, D., Chen, W., & Yang, D. (2021) · doi:10.18653/v1/2021.acl-long.338 · Cited by 35 · see citing works
Gradient-based Parameter Selection for Efficient Fine-Tuning
Zhang, Z., Zhang, Q., Gao, Z., Zhang, R., Shutova, E., Zhou, S., & Zhang, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02699 · Cited by 33 · see citing works
Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningPreprint
Xu, R., Luo, F., Zhang, Z., Tan, C., Chang, B., Huang, S., & Huang, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05687 · Cited by 33 · see citing works
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
Guo, C., Cheng, F., Du, Z., Kiessling, J., Ku, J., Li, S., Li, Z., Ma, M., Molom-Ochir, T., Morris, B., et al. (2025) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2024.3476008 · Cited by 32 · see citing works
APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models
Wang, Q., Mao, Y., Wang, J., Yu, H., Nie, S., Wang, S., Feng, F., Huang, L., Quan, X., Xu, Z., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.567 · Cited by 31 · see citing works
AutoProSAM: Automated Prompting SAM for 3D Multi-Organ Segmentation
Li, C., Sultan, R. I., Khanduri, P., Qiang, Y., Chetty, I. J., & Zhu, D. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00352 · Cited by 31 · see citing works
XPrompt: Exploring the Extreme of Prompt Tuning
Ma, F., Zhang, C., Ren, L., Wang, J., Wang, Q., Wu, W., Quan, X., & Song, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.758 · Cited by 31 · see citing works
Efficient LLMs Training and Inference: An Introduction
Li, R., Fu, D., Shi, C., Huang, Z., & Lu, G. (2024) · IEEE Access · doi:10.1109/access.2024.3501358 · Cited by 30 · see citing works
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting
Yong, Z. X., Schoelkopf, H., Muennighoff, N., Aji, A. F., Adelani, D. I., Almubarak, K., Bari, M. S., Sutawika, L., Kasai, J., Baruwa, A., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.653 · Cited by 30 · see citing works
Prompt tuning for parameter-efficient medical image segmentation
Fischer, M., Bartler, A., & Yang, B. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103024 · Cited by 30 · see citing works
Hydra: Multi-head low-rank adaptation for parameter efficient fine-tuning
Kim, S., Yang, H.-M., Kim, Y., Hong, Y., & Park, E. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106414 · Cited by 28 · see citing works
Self-citationLearning Expressive Prompting With Residuals for Vision Transformers
Das, R., Dukler, Y., Ravichandran, A., & Swaminathan, A. D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00328 · Cited by 24 · see citing works
Event-Adapted Video Super-Resolution
Xiao, Z., Kai, D., Zhang, Y., Zha, Z.-J., Sun, X., & Xiong, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72946-1_13 · Cited by 24 · see citing works
FLASC: Federated LoRA with Sparse Communication
Kuo, K., Raje, A., Rajesh, K., & Smith, V. (2026) · ACM Conference on AI and Agentic Systems (CAIS) · doi:10.1145/3786335.3813151 · Cited by 23 · see citing works
Efficient Test Time Adapter Ensembling for Low-resource Language Varieties
Wang, X., Tsvetkov, Y., Ruder, S., & Neubig, G. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.63 · Cited by 23 · see citing works
Universal Deep Image Compression via Content-Adaptive Optimization with Adapters
Tsubota, K., Akutsu, H., & Aizawa, K. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00256 · Cited by 23 · see citing works
MixPAVE: Mix-Prompt Tuning for Few-shot Product Attribute Value Extraction
Yang, L., Wang, Q., Wang, J., Quan, X., Feng, F., Chen, Y., Khabsa, M., Wang, S., Xu, Z., & Liu, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.633 · Cited by 23 · see citing works
Jafari, M., Tao, X., Barua, P. D., Tan, R., & Acharya, U. R. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102982 · Cited by 22 · see citing works
Densely Connected Parameter-Efficient Tuning for Referring Image Segmentation
Huang, J., Xu, Z., Liu, T., Liu, Y., Han, H., Yuan, K., & Li, X. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i4.32380 · Cited by 22 · see citing works
Lin, J., Zhang, K., Zhang, L., Liu, P., Peng, W., Zhang, H., Yan, X., Liu, C., Yang, Y., Sun, H. X., et al. (2023) · Fuel · doi:10.1016/j.fuel.2023.129343 · Cited by 22 · see citing works
Transfer learning for language model adaptation
Saiful, B. M. (2023) · doi:10.32657/10356/169892 · Cited by 22 · see citing works
Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models
Lawton, N., Kumar, A., Thattai, G., Galstyan, A., & Steeg, G. V. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.539 · Cited by 19 · see citing works
Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems
Varshney, N., & Baral, C. R. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.756 · Cited by 19 · see citing works
UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory
Diao, H., Wan, B., Zhang, Y., Jia, X., Lu, H., & Chen, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02714 · Cited by 19 · see citing works
Learning Kernel-Smoothed Machine Translation with Retrieved Examples
Jiang, Q., Wang, M., Cao, J., Cheng, S., Huang, S., & Li, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.579 · Cited by 18 · see citing works
Thin, D. V., Hao, D. N., & Nguyen, N. L.-T. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3589131 · Cited by 18 · see citing works
Structure-Aware Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
Hu, Y., Xie, Y., Wang, T., Chen, M., & Pan, Z. (2023) · Mathematics · doi:10.3390/math11204317 · Cited by 18 · see citing works
MACP: Efficient Model Adaptation for Cooperative Perception
Ma, Y., Lu, J., Cui, C., Zhao, S., Cao, X., Ye, W., & Wang, Z. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00334 · Cited by 18 · see citing works
Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation
Hou, Z., Salazar, J., & Polovets, G. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00517 · Cited by 17 · see citing works
Parameter-Efficient Sparsity for Large Language Models Fine-Tuning
Li, Y., Luo, F., Tan, C., Wang, M., Huang, S., Li, S., & Bai, J. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/586 · Cited by 16 · see citing works
Fan, Z., Zhang, Q., Abillama, P., Shoouri, S., Lee, C. S. G., Blaauw, D. T., Kim, H.-S., & Sylvester, D. (2023) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3579371.3589040 · Cited by 16 · see citing works
Advancing Parameter Efficiency in Fine-tuning via Representation Editing
Wu, M., Liu, W., Wang, X., Li, T., Lv, C., Ling, Z., JianHao, Z., Zhang, C., Zheng, X., & Huang, X. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.726 · Cited by 15 · see citing works
LLMs on a Budget: System-Level Approaches to Power-Efficient and Scalable Fine-Tuning
Gogineni, K., Suvizi, A., & Venkataramani, G. (2025) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2025.3580498 · Cited by 14 · see citing works
Beyond Classification: Inferring Function Names in Stripped Binaries via Domain Adapted LLMs
Jiang, L., Jin, X., & Lin, Z. (2025) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2025.240797 · Cited by 14 · see citing works
PEDRO: Parameter-Efficient Fine-Tuning with Prompt DEpenDent Representation MOdification
Xie, T., Li, T., Zhu, W., & Han, W. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-6603-4_18 · Cited by 14 · see citing works
MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension
Liu, T., Xu, Z., Hu, Y., Shi, L., Wang, Z., & Yin, Q. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.287 · Cited by 13 · see citing works
Foundation Model Engineering: Engineering Foundation Models Just as Engineering Software
Ran, D., Wu, M. S., Yang, W. M., & Xie, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3719005 · Cited by 12 · see citing works
Experimental Evaluation of Parameter-Efficient Fine-Tuning for Software Engineering Tasks
Zou, W., Shen, Z., Li, Q., Ge, J., Li, C., Chen, X., Shen, X., Huang, L., & Luo, B. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3722107 · Cited by 12 · see citing works
Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modeling
Shim, K., Choi, I., Sung, W., & Choi, J. (2021) · International SoC Design Conference · doi:10.1109/isocc53507.2021.9613933 · Cited by 12 · see citing works
Personalized LoRA for Human-Centered Text Understanding
Zhang, Y., Wang, J., Yu, L., Xu, D., & Zhang, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29931 · Cited by 12 · see citing works
Facial Affective Behavior Analysis with Instruction Tuning
Li, Y., Dao, A., Bao, W., Tan, Z., Chen, T., Liu, H., & Kong, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72649-1_10 · Cited by 11 · see citing works
Wang, Z., Chen, D., Dai, D., Xu, R., Li, Z., & Wu, Y. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.46 · Cited by 11 · see citing works
He, Y., Zhu, X., Li, D., & Wang, H. (2025) · Electronics · doi:10.3390/electronics14101961 · Cited by 11 · see citing works
Modular and On-demand Bias Mitigation with Attribute-Removal Subnetworks
Hauzenberger, L., Masoudian, S., Kumar, D., Schedl, M., & Rekabsaz, N. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.386 · Cited by 11 · see citing works
Non-Parametric Domain Adaptation for End-to-End Speech Translation
Du, Y., Wang, W., Zhang, Z., Chen, B., Xu, T. W., Xie, J., & Chen, E. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.21 · Cited by 10 · see citing works
SMoP: Towards Efficient and Effective Prompt Tuning with Sparse Mixture-of-Prompts
Choi, J., Kim, J., Park, J.-H., Mok, W.-L., & Lee, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.884 · Cited by 10 · see citing works
Spadaro, G., Renzulli, R., Bragagnolo, A., Giraldo, J. H., Fiandrotti, A., Grangetto, M., & Tartaglione, E. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00165 · Cited by 10 · see citing works
Xu, L., Yuan, S., Wang, D., Chen, Y., Li, X., & Sun, Y. (2024) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3649329.3657327 · Cited by 9 · see citing works
- Authors
- Guo, D., Rush, A., & Kim, Y.
- Affiliations
- Harvard University (US)
- Cornell University (US)
- IBM (United States) (US)
- Venue
- Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- Published
- 2021
- Pages
- 4884-4896
- OpenAlex
- W3174702398
- Cited by
- 220
- References
- 89
- Citations by year
- 2026: 22 · 2025: 42 · 2024: 51 · 2023: 63 · 2022: 35 · 2021: 7
- FWCI
- 18.02
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Funders
- National Science Foundation (1704834, 2037519)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- parameter-efficient transfer learning, diffusion models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
