TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models

Attention Is All You Need

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works

Transformers: State-of-the-Art Natural Language Processing

Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works

Unsupervised Cross-lingual Representation Learning at Scale

Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637 · see citing works

Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity RecognitionPreprint

Sang, E. T. K. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0209010 · Cited by 1,677 · see citing works

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Frankle, J., & Carbin, M. (2018) · arXiv · doi:10.48550/arxiv.1803.03635 · Cited by 1,529 · see citing works

Learning both Weights and Connections for Efficient Neural Networks

Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481 · see citing works

Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146 · see citing works

XNLI: Evaluating Cross-lingual Sentence Representations

Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S. R., Schwenk, H., & Stoyanov, V. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1269 · Cited by 949 · see citing works

To prune, or not to prune: exploring the efficacy of pruning for model compression

Zhu, M., & Gupta, S. (2017) · arXiv · doi:10.48550/arxiv.1710.01878 · Cited by 805 · see citing works

Movement pruning: Adaptive sparsity by fine-tuning

Sanh, V., Wolf, T., & Rush, A. M. (2020) · arXiv · doi:10.48550/arxiv.2005.07683 · Cited by 218 · see citing works

Block Pruning For Faster Transformers

Lagunas, F., Charlaix, E., Sanh, V., & Rush, A. M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.829 · Cited by 144 · see citing works

DynaBERT: Dynamic BERT with Adaptive Width and Depth

Hou, L., Huang, Z., Shang, L., Jiang, X., Chen, X., & Liu, Q. (2020) · arXiv · doi:10.48550/arxiv.2004.04037 · Cited by 143 · see citing works

Load What You Need: Smaller Versions of Mutililingual BERT

Abdaoui, A., Pradel, C., & Sigel, G. (2020) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2020.sustainlp-1.16 · Cited by 53 · see citing works

TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing

Yang, Z., Cui, Y., Chen, Z., Che, W., Liu, T., Wang, S., & Hu, G. (2020) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2020.acl-demos.2 · Cited by 34 · see citing works

Multilingual multi-aspect explainability analyses on machine reading comprehension models

Cui, Y., Zhang, W.-N., Che, W., Liu, T., Chen, Z., & Wang, S. (2022) · iScience · doi:10.1016/j.isci.2022.104176 · Cited by 11 · see citing works

Survey of Different Large Language Model Architectures: Trends, Benchmarks, and Challenges

Shao, M., Basit, A., Karri, R., & Shafique, M. (2024) · IEEE Access · doi:10.1109/access.2024.3482107 · Cited by 110 · see citing works

RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs

Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92 · see citing works

Applications of Pruning Methods in Natural Language Processing

Touheed, M., Zubair, U., Sabir, D., Hassan, A., Butt, M. F. U., Riaz, F., Abdul, W., & Ayub, R. (2024) · IEEE Access · doi:10.1109/access.2024.3411776 · Cited by 13 · see citing works

BMCook: A Task-agnostic Compression Toolkit for Big Models

Zhang, Z., Gong, B., Chen, Y., Han, X., Zeng, G., Zhao, W., Chen, Y., Liu, Z., & Sun, M. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.40 · Cited by 8 · see citing works

POP-VQA – Privacy preserving, On-device, Personalized Visual Question Answering

Sahu, P. P., Raut, A., Samant, J. S., Gorijala, M., Lakshminarayanan, V., & Bhaskar, P. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00828 · Cited by 6 · see citing works

NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models

Ko, J., Park, S., Kim, Y., Ahn, S., Chang, D.-S., Ahn, E., & Yun, S.-Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.404 · Cited by 5 · see citing works

Legal-Tech Open Diaries: Lesson learned on how to develop and deploy light-weight models in the era of humongous Language Models

Maroudas, S., Legkas, S., Malakasiotis, P., & Chalkidis, I. (2022) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2022.nllp-1.8 · Cited by 4 · see citing works

Ten Key Problems of Pre-trained Models: An Outlook of Representation Learning

Ding, N., Chen, W., Zhang, Z., Hu, S., Cui, G., Yao, Y., Qin, Y., Zeng, Z., Han, X., Liu, Z., et al. (2023) · Springer eBooks · doi:10.1007/978-981-99-1600-9_14 · Cited by 3 · see citing works

Improving Non-Autoregressive Translation Quality With Pretrained Language Model, Embedding Distillation and Upsampling Strategy for CTC

Syu, S.-S., Xie, J., & Lee, H.-Y. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3451977 · Cited by 4 · see citing works

MUX-PLMs: Data Multiplexing for High-throughput Language Models

Murahari, V., Deshpande, A., Jimenez, C., Izhak Shafran, J., Wang, M., Cao, Y., & Narasimhan, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.301 · Cited by 3 · see citing works

MUX-PLMs: Pre-training Language Models with Data Multiplexing

Murahari, V., Deshpande, A., Jimenez, C., Izhak Shafran, J., Wang, M., Cao, Y., & Narasimhan, K. (2023) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2023.repl4nlp-1.17 · Cited by 2 · see citing works

Explainable Structured Pruning of BERT via Mutual Information

Huang, H., Song, H.-J., & Zhao, Q. (2025) · Entropy · doi:10.3390/e27121224 · Cited by 1 · see citing works

Compression Strategies for Massive Natural Language Generation Models

Dayal, A., Angara, J., Gupta, S., & Saripalle, R. S. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-77081-4_7

HeadSkip: Characterizing and Accelerating ‘Small’ Language Models on Edge CPUs

Sartor, I., Ganesan, K., Nguyen, A., & Moshovos, A. (2026) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass69572.2026.00053

LiteOCR: A Lightweight Text Detection Framework Integrating Cross-Modal Semantic Distillation and Dynamic Resolution Decision-Making

Mao, D.-H., Xie, M.-D., zhao, Z.-H., & Fang, F. (2026) · 電腦學刊 · doi:10.63367/199115992026083704005

Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling

Liu, L., Ren, X., Shang, J., Gu, X., Peng, J., & Han, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1153 · Cited by 45 · see citing works

From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression

Xu, R., Luo, F., Wang, C., Chang, B., Huang, J. G., Huang, S., & Huang, F. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21408 · Cited by 22 · see citing works

Task-specific Compression for Multi-task Language Models using Attribution-based Pruning

Yang, N., Jang, Y., Lee, H., Jeong, S., & Jung, K. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.43 · Cited by 11 · see citing works

IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model PretrainingPreprint

Li, Y., Du, X., Jaiswal, A., Lei, T., Zhao, T., Wang, C., & Wang, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.05920

Compact Language Models via Pruning and Knowledge Distillation

Muralidharan, S., Sreenivas, S. T., Joshi, R., Chochowski, M., Patwary, M. A., Shoeybi, M., Catanzaro, B., Kautz, J., & Molchanov, P. (2024) · neural information processing systems · doi:10.52202/079017-1299 · Cited by 53 · see citing works