The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Open accessAbstract
We develop a novel method, called PoWER-BERT, for improving the inference time of the popular BERT model, while maintaining the accuracy. It works by: a) exploiting redundancy pertaining to word-vectors (intermediate encoder outputs) and eliminating the redundant vectors. b) determining which word-vectors to eliminate by developing a strategy for measuring their significance, based on the self-attention mechanism. c) learning how many word-vectors to eliminate by augmenting the BERT model and the loss function. Experiments on the standard GLUE benchmark shows that PoWER-BERT achieves up to 4.5x reduction in inference time over BERT with <1% loss in accuracy. We show that PoWER-BERT offers significantly better trade-off between accuracy and inference time compared to prior methods. We demonstrate that our method attains up to 6.8x reduction in inference time with <1% loss in accuracy when applied over ALBERT, a highly compressed version of BERT. The code for PoWER-BERT is publicly available at https://github.com/IBM/PoWER-BERT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“(1) Token pruning (Goyal et al., 2020; Kim and Cho, 2021; Modarressi et al., 2022) and token merging (Bolya et al., 2023), which need model fine-tuning or intermediate results during inference and have been used with BERT-scale models.”
From LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2024) · cites this paper as Goyal et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Research works printed but not linked: 26
- ¸alo M. Correia, Vlad Niculae, and André F. T. Martins. Adaptively sparse transformers. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.
- Misha Denil, Babak Shakibi, Laurent Dinh, Marc'Aurelio Ranzato, and Nando De Freitas. Predicting parameters in deep learning. In NIPS, 2013.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional trans- formers for language understanding. In NAACL, 2019.
- Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bour- dev. Compressing deep convolutional networks using vec- tor quantization. arXiv preprint arXiv:1412.6115, 2014. · arXiv
- Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NIPS, 2015.
- Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In ICCV, 2017.
- Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distill- ing the knowledge in a neural network. arxiv preprint: 1503.02531, 2015. · arXiv
- Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compression of deep con- volutional neural networks for fast and low power mobile applications. arxiv preprint: 1511.06530, 2015. · arXiv
- Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading com- prehension dataset from examinations. arXiv preprint arXiv:1704.04683, 2017. · arXiv
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language repre- sentations. arXiv preprint arXiv:1909.11942, 2019. · arXiv
- Yang Liu and Mirella Lapata. Text summarization with pre- trained encoders. arXiv preprint arXiv:1908.08345, 2019. · arXiv
- Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arxiv preprint: 1904.09482, 2019. · arXiv
- Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, 2011.
- J. S. McCarley. Pruning a BERT-based question answering model. arxiv preprint: 1910.06360, 2019. · arXiv
- Paul Michel, Omer Levy, and Graham Neubig. Are six- teen heads really better than one? arXiv preprint arXiv:1905.10650, 2019. · arXiv
- Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arxiv preprint: 1611.06440, 2016. · arXiv
- Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. arxiv preprint: 1701.05369, 2017. · arXiv
- Ben Peters, Vlad Niculae, and André F. T. Martins. Sparse sequence-to-sequence models. Proceedings of the 57th An- nual Meeting of the Association for Computational Lin- guistics, 2019.
- Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. · arXiv
- Bharat Bhusan Sau and Vineeth N. Balasubramanian. Deep model compression: Distilling knowledge from noisy teachers. arxiv preprint: 1610.09650, 2016. · arXiv
- Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Q-BERT: Hessian based ultra low precision quantization of BERT. arxiv preprint:1909.05840, 2019. · arXiv
- Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Pa- tient knowledge distillation for BERT model compression. arXiv preprint arXiv:1908.09355, 2019. · arXiv
- Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi- task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
- Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Struc- tured pruning of large language models. arxiv preprint: 1910.04732, 2019. · arXiv
- Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8BERT: Quantized 8bit BERT. arxiv preprint:1910.06188, 2019. · arXiv
- Guangxiang Zhao, Junyang Lin, Zhiyuan Zhang, Xuancheng Ren, Qi Su, and Xu Sun. Explicit sparse trans- former: Concentrated attention through explicit selection. arXiv,1912.11637, 2019. · arXiv
Cited by
Sorted by citations
Patch Slimming for Efficient Vision Transformers
Tang, Y., Han, K., Wang, Y., Xu, C., Guo, J., Xu, C., & Tao, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01185 · Cited by 201
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
Ganesh, P., Chen, Y., Lou, X., Khan, M. A., Yang, Y., Sajjad, H., Nakov, P., Chen, D., & Winslett, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00413 · Cited by 190
A Review on Edge Large Language Models: Design, Execution, and Applications
Zheng, Y., Chen, Y., Qian, B., Shi, X., Shu, Y., & Chen, J. (2025) · ACM Computing Surveys · doi:10.1145/3719664 · Cited by 169
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., & Qiu, L. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.825 · Cited by 168
Token Merging: Your ViT But Faster
Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C., & Hoffman, J. (2022) · arXiv · doi:10.48550/arxiv.2210.09461 · Cited by 140
Metadata
- Authors
- Goyal, S., Choudhury, A. R., Raje, S. M., Chakaravarthy, V. T., Sabharwal, Y., & Verma, A.
- Affiliations
- IBM Research - India (IN)
- IBM (United States) (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Volume
- 1
- Pages
- 3690-3699
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2001.08950
- OpenAlex
- W3034742519
- Cited by
- 117
- References
- 1
- Citations by year
- 2026: 3 · 2025: 16 · 2024: 28 · 2023: 29 · 2022: 21 · 2021: 15 · 2020: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- BERT, inference acceleration, self-attention, redundancy reduction, model compression, ALBERT, inference latency reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
