CoChat
Sign Up
Cited by 42Advanced Neural Network ApplicationsTopic ModelingNatural Language Processing Techniques

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

TPrune: Efficient Transformer Pruning for Mobile Devices

Mao, Jiachen, Yang, Huanrui, Li, Ang, Li, Hai, Chen, Yiran·2021·ACM Transactions on Cyber-Physical Systems
DOI 10.1145/3446640
Open this paper in CoChat — free account

Abstract

The invention of Transformer model structure boosts the performance of Neural Machine Translation (NMT) tasks to an unprecedented level. Many previous works have been done to make the Transformer model more execution-friendly on resource-constrained platforms. These researches can be categorized into three key fields: Model Pruning, Transfer Learning, and Efficient Transformer Variants. The family of model pruning methods are popular for their simplicity in practice and promising compression rate and have achieved great success in the field of convolution neural networks (CNNs) for many vision tasks. Nonetheless, previous Transformer pruning works did not perform a thorough model analysis and evaluation on each Transformer component on off-the-shelf mobile devices. In this work, we analyze and prune transformer models at the line-wise granularity and also implement our pruning method on real mobile platforms. We explore the properties of all Transformer components as well as their sparsity features, which are leveraged to guide Transformer model pruning. We name our whole Transformer analysis and pruning pipeline as TPrune. In TPrune, we first propose Block-wise Structured Sparsity Learning (BSSL) to analyze Transformer model property. Then, based on the characters derived from BSSL, we apply Structured Hoyer Square (SHS) to derive the final pruned models. Comparing with the state-of-the-art Transformer pruning methods, TPrune is able to achieve a higher model compression rate with less performance degradation. Experimental results show that our pruned models achieve 1.16×–1.92× speedup on mobile devices with 0%–8% BLEU score degradation compared with the original Transformer model.

Overview

Integrity signals

Registry

Crossref · 10.1145/3446640

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Flagged first, then by citations

Show all 3 references

Research works printed but not linked: 42

  1. Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , Łukasz Kaiser , and Illia Polosukhin . 2017. Attention is all you need . In Proceedings of the International Conference on Advances in Neural Information Processing Systems , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc. , 5998--6008. Retrieved from http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the International Conference on Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 5998--6008. Retrieved from http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf.
  2. Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2018 . BERT: Pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018). Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018). · arXiv
  3. Zhilin Yang , Zihang Dai , Yiming Yang , Jaime Carbonell , Russ R. Salakhutdinov , and Quoc V . Le . 2019 . XLNet: Generalized autoregressive pretraining for language understanding. In Proceedings of the International Conference on Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 5754--5764. Retrieved from http://papers.nips.cc/paper/8812-xlnet-generalized-autoregressive-pretraining-for-language-understanding.pdf. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Proceedings of the International Conference on Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 5754--5764. Retrieved from http://papers.nips.cc/paper/8812-xlnet-generalized-autoregressive-pretraining-for-language-understanding.pdf.
  4. Yinhan Liu , Myle Ott , Naman Goyal , Jingfei Du , Mandar Joshi , Danqi Chen , Omer Levy , Mike Lewis , Luke Zettlemoyer , and Veselin Stoyanov . 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR abs/1907.11692 ( 2019 ). Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019). · arXiv
  5. 10.1145/3316781.3317865 · DOI
  6. 10.1145/3240765.3240851 · DOI
  7. 10.1145/3316781.3317936 · DOI
  8. J. Mao , Z. Qin , Z. Xu , K. W. Nixon , X. Chen , H. Li , and Y. Chen . 2017. AdaLearner: An adaptive distributed mobile learning system for neural networks . In Proceedings of the IEEE/ACM International Conference on Computer-Aided Design (ICCAD’17) . 291--296. DOI:http://dx.doi.org/10.1109/ICCAD. 2017 .8203791 J. Mao, Z. Qin, Z. Xu, K. W. Nixon, X. Chen, H. Li, and Y. Chen. 2017. AdaLearner: An adaptive distributed mobile learning system for neural networks. In Proceedings of the IEEE/ACM International Conference on Computer-Aided Design (ICCAD’17). 291--296. DOI:http://dx.doi.org/10.1109/ICCAD.2017.8203791 · DOI
  9. 10.1109/ASPDAC.2018.8297378 · DOI
  10. 10.1109/HPCA.2019.00027 · DOI
  11. 10.1145/3287624.3287642 · DOI
  12. Song Han , Huizi Mao , and William J . Dally . 2015 . Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. arXiv preprint arXiv:1510.00149 (2015). Song Han, Huizi Mao, and William J. Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. arXiv preprint arXiv:1510.00149 (2015). · arXiv
  13. Wei Wen , Chunpeng Wu , Yandan Wang , Yiran Chen , and Hai Li . 2016 . Learning structured sparsity in deep neural networks . In Proceedings of the International Conference on Advances in Neural Information Processing Systems, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.). Curran Associates, Inc. , 2074--2082. Retrieved from http://papers.nips.cc/paper/6504-learning-structured-sparsity-in-deep-neural-networks.pdf. Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. 2016. Learning structured sparsity in deep neural networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.). Curran Associates, Inc., 2074--2082. Retrieved from http://papers.nips.cc/paper/6504-learning-structured-sparsity-in-deep-neural-networks.pdf.
  14. Wei Wen , Yuxiong He , Samyam Rajbhandari , Minjia Zhang , Wenhan Wang , Fang Liu , Bin Hu , Yiran Chen , and Hai Li. 2017. Learning intrinsic sparse structures within long short-term memory. arXiv preprint arXiv:1709.05027 ( 2017 ). Wei Wen, Yuxiong He, Samyam Rajbhandari, Minjia Zhang, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, and Hai Li. 2017. Learning intrinsic sparse structures within long short-term memory. arXiv preprint arXiv:1709.05027 (2017). · arXiv
  15. Qing Yang , Jiachen Mao , Zuoguan Wang , and Hai Li. 2019. DASNet: Dynamic activation sparsity for neural network efficiency improvement. arXiv preprint arXiv:1909.06964 ( 2019 ). Qing Yang, Jiachen Mao, Zuoguan Wang, and Hai Li. 2019. DASNet: Dynamic activation sparsity for neural network efficiency improvement. arXiv preprint arXiv:1909.06964 (2019). · arXiv
  16. Mengye Ren et al . 2018 . SBNet: Sparse blocks network for fast inference . In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’18) . 8711--8720. Mengye Ren et al. 2018. SBNet: Sparse blocks network for fast inference. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’18). 8711--8720.
  17. Mikhail Figurnov et al . 2016 . PerforatedCNNs: Acceleration through elimination of redundant convolutions . In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 947--955 . Mikhail Figurnov et al. 2016. PerforatedCNNs: Acceleration through elimination of redundant convolutions. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 947--955.
  18. Angela Fan , Edouard Grave , and Armand Joulin . 2019. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv:1909.11556 ( 2019 ). Angela Fan, Edouard Grave, and Armand Joulin. 2019. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv:1909.11556 (2019). · arXiv
  19. Elena Voita , David Talbot , Fedor Moiseev , Rico Sennrich , and Ivan Titov . 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418 ( 2019 ). Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418 (2019). · arXiv
  20. Paul Michel , Omer Levy , and Graham Neubig . 2019 . Are sixteen heads really better than one? In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 14014--14024 . Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 14014--14024.
  21. Kenton Murray , Jeffery Kinnison , Toan Q. Nguyen , Walter Scheirer , and David Chiang . 2019. Auto-sizing the transformer network: Improving speed, efficiency, and performance for low-resource machine translation. arXiv preprint arXiv:1910.06717 ( 2019 ). Kenton Murray, Jeffery Kinnison, Toan Q. Nguyen, Walter Scheirer, and David Chiang. 2019. Auto-sizing the transformer network: Improving speed, efficiency, and performance for low-resource machine translation. arXiv preprint arXiv:1910.06717 (2019). · arXiv
  22. Victor Sanh , Lysandre Debut , Julien Chaumond , and Thomas Wolf . 2019. DistilBERT , a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 ( 2019 ). Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019). · arXiv
  23. Zhiqing Sun , Hongkun Yu , Xiaodan Song , Renjie Liu , Yiming Yang , and Denny Zhou . 2020 . Mobilebert: A compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.02984. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. Mobilebert: A compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.02984. · arXiv
  24. Biao Zhang , Deyi Xiong , and Jinsong Su. 2018. Accelerating neural transformer via an average attention network. arXiv preprint arXiv:1805.00631 ( 2018 ). Biao Zhang, Deyi Xiong, and Jinsong Su. 2018. Accelerating neural transformer via an average attention network. arXiv preprint arXiv:1805.00631 (2018). · arXiv
  25. Zhanghao Wu , Zhijian Liu , Ji Lin , Yujun Lin , and Song Han . 2020 . Efficient transformer for mobile applicatoins . In Proceedings of the International Conference on Learning Representitive (ICLR’20) . Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. 2020. Efficient transformer for mobile applicatoins. In Proceedings of the International Conference on Learning Representitive (ICLR’20).
  26. Yonghui Wu , Mike Schuster , Zhifeng Chen , Quoc V. Le , Mohammad Norouzi , Wolfgang Macherey , Maxim Krikun , Yuan Cao , Qin Gao , Klaus Macherey et al . 2016 . Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016). Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016). · arXiv
  27. Christian Szegedy , Sergey Ioffe , Vincent Vanhoucke , and Alexander A. Alemi . 2017. Inception-v4, Inception-ResNet and the impact of residual connections on learning . In Proceedings of the 31st AAAI Conference on Artificial Intelligence. Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A. Alemi. 2017. Inception-v4, Inception-ResNet and the impact of residual connections on learning. In Proceedings of the 31st AAAI Conference on Artificial Intelligence.
  28. Alex Krizhevsky , Geoffrey Hinton et al . 2009 . Learning multiple layers of features from tiny images. Citeseer . Alex Krizhevsky, Geoffrey Hinton et al. 2009. Learning multiple layers of features from tiny images. Citeseer.
  29. Karen Simonyan and Andrew Zisserman . 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 ( 2014 ). Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014). · arXiv
  30. Jie Hao , Xing Wang , Shuming Shi , Jinfeng Zhang , and Zhaopeng Tu. 2019. Multi-granularity self-attention for neural machine translation. arXiv preprint arXiv:1909.02222 ( 2019 ). Jie Hao, Xing Wang, Shuming Shi, Jinfeng Zhang, and Zhaopeng Tu. 2019. Multi-granularity self-attention for neural machine translation. arXiv preprint arXiv:1909.02222 (2019). · arXiv
  31. Ziheng Wang , Jeremy Wohlwend , and Tao Lei . 2019. Structured pruning of large language models. arXiv preprint arXiv:1910.04732 ( 2019 ). Ziheng Wang, Jeremy Wohlwend, and Tao Lei. 2019. Structured pruning of large language models. arXiv preprint arXiv:1910.04732 (2019). · arXiv
  32. J. S. McCarley . 2019. Pruning a BERT-based question answering model. arXiv preprint arXiv:1910.06360 ( 2019 ). J. S. McCarley. 2019. Pruning a BERT-based question answering model. arXiv preprint arXiv:1910.06360 (2019). · arXiv
  33. Fu-Ming Guo , Sijia Liu , Finlay S. Mungall , Xue Lin , and Yanzhi Wang . 2019. Reweighted proximal pruning for large-scale language representation. arXiv preprint arXiv:1909.12486 ( 2019 ). Fu-Ming Guo, Sijia Liu, Finlay S. Mungall, Xue Lin, and Yanzhi Wang. 2019. Reweighted proximal pruning for large-scale language representation. arXiv preprint arXiv:1909.12486 (2019). · arXiv
  34. Sheng Shen , Zhen Dong , Jiayu Ye , Linjian Ma , Zhewei Yao , Amir Gholami , Michael W. Mahoney , and Kurt Keutzer . 2019. Q-BERT: Hessian based ultra low precision quantization of BERT. arXiv preprint arXiv:1909.05840 ( 2019 ). Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2019. Q-BERT: Hessian based ultra low precision quantization of BERT. arXiv preprint arXiv:1909.05840 (2019). · arXiv
  35. Ofir Zafrir , Guy Boudoukh , Peter Izsak , and Moshe Wasserblat . 2019. Q8BERT: Quantized 8bit BERT. arXiv preprint arXiv:1910.06188 ( 2019 ). Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8BERT: Quantized 8bit BERT. arXiv preprint arXiv:1910.06188 (2019). · arXiv
  36. Song Han , Jeff Pool , John Tran , and William Dally . 2015 . Learning both weights and connections for efficient neural network . In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 1135--1143 . Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 1135--1143.
  37. Baoyuan Liu , Min Wang , Hassan Foroosh , Marshall Tappen , and Marianna Pensky . 2015 . Sparse convolutional neural networks . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 806--814 . Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. 2015. Sparse convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 806--814.
  38. Huanrui Yang , Wei Wen , and Hai Li. 2019. DeepHoyer: Learning sparser neural network with differentiable scale-invariant sparsity measures. arXiv preprint arXiv:1908.09979 ( 2019 ). Huanrui Yang, Wei Wen, and Hai Li. 2019. DeepHoyer: Learning sparser neural network with differentiable scale-invariant sparsity measures. arXiv preprint arXiv:1908.09979 (2019). · arXiv
  39. 10.18653/v1/P18-1167 · DOI
  40. Martín Abadi et al . 2016 . Tensorflow: A system for large-scale machine learning . In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI’16) . 265--283. Martín Abadi et al. 2016. Tensorflow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI’16). 265--283.
  41. Ashish Vaswani , Samy Bengio , Eugene Brevdo , Francois Chollet , Aidan N. Gomez , Stephan Gouws , Llion Jones , Łukasz Kaiser , Nal Kalchbrenner , Niki Parmar , Ryan Sepassi , Noam Shazeer , and Jakob Uszkoreit . 2018. Tensor2Tensor for neural machine translation. CoRR abs/1803.07416 ( 2018 ). Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2Tensor for neural machine translation. CoRR abs/1803.07416 (2018). · arXiv
  42. 10.2478/pralin-2018-0002 · DOI

Cited by

Sorted by citations

See all 42

Metadata

Authors
Mao, J., Yang, H., Li, A., Li, H., & Chen, Y.
Affiliations
Duke University (US)
ORCID
Venue
ACM Transactions on Cyber-Physical Systems
Publisher
Association for Computing Machinery (ACM)
Published
2021-04-15
Volume
5
Issue
3
Pages
1-22
ISSN
  • 2378-962X (ISSN-L)
  • 2378-9638
DOI
10.1145/3446640
OpenAlex
W3154851733
Cited by
42
References
3
Citations by year
2026: 4 · 2025: 7 · 2024: 9 · 2023: 13 · 2022: 7 · 2021: 2
FWCI
2.79
Type
journal article
Language
en
Open access
closed
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
ACM Transactions on Cyber-Physical Systems (publishedVersion)
Funders
National Science Foundation (CNS-1717657, CCF-1937435, CNS-1822085)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
neural machine translation, model compression, structured sparsity, mobile devices, resource-constrained platforms

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review