CoChat
Sign Up
Cited by 23Topic ModelingExplainable Artificial Intelligence (XAI)Advanced Neural Network Applications

AdapLeR: Speeding up Inference by Adaptive Length Reduction

Open access
Modarressi, Ali, Mohebbi, Hosein, Pilehvar, Mohammad Taher·2022·Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
DOI 10.18653/v1/2022.acl-long.1
Open this paper in CoChat — free accountOpen PDF

Abstract

Pre-trained language models have shown stellar performance in various downstream tasks.But, this usually comes at the cost of high latency and computation, hindering their usage in resource-limited settings.In this work, we propose a novel approach for reducing the computational cost of BERT with minimal loss in downstream performance.Our method dynamically eliminates less contributing tokens through layers, resulting in shorter lengths and consequently lower computational cost.To determine the importance of each token representation, we train a Contribution Predictor for each layer using a gradient-based saliency method.Our experiments on several diverse classification tasks show speedups up to 22x during inference time without much sacrifice in performance.We also validate the quality of the selected tokens in our method using human annotations in the ERASER benchmark.In comparison to other widely used strategies for selecting important tokens, such as saliency and attention, our proposed method has a significantly lower false positive rate in generating rationales.Our code is freely available at https://github.com/amodaresi/ AdapLeR.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2022.acl-long.1

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Token pruning and token merging methods need model fine-tuning or intermediate results during inference and have been used with BERT-scale models.

    “(1) Token pruning (Goyal et al., 2020; Kim and Cho, 2021; Modarressi et al., 2022) and token merging (Bolya et al., 2023), which need model fine-tuning or intermediate results during inference and have been used with BERT-scale models.”

    From LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2024) · cites this paper as Modarressi et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 33

  1. Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. 2018. Towards better understanding of gradient-based attribution methods for deep neural networks. In International Conference on Learning Representations.
  2. William B. Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
  3. Cristóbal Eyzaguirre, Felipe del Río, Vladimir Araujo, and Álvaro Soto. 2021. DACT-BERT: Differentiable adaptive computation time for an efficient bert infer- ence. arXiv preprint arXiv:2109.11745. · arXiv
  4. Yunchao Gong, L. Liu, Ming Yang, and Lubomir D. Bourdev. 2014. Compressing deep convolu- tional networks using vector quantization. ArXiv, abs/1412.6115. · arXiv
  5. Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, and Ashish Verma. 2020. Power-bert: Accelerating bert inference via progressive word-vector elimina- tion. In International Conference on Machine Learn- ing, pages 3690-3699. PMLR.
  6. Song Han, Huizi Mao, and William J. Dally. 2016. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In 4th International Conference on Learning Represen- tations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
  7. Yihui He, Xiangyu Zhang, and Jian Sun. 2017. Channel pruning for accelerating very deep neural networks. 2017 IEEE International Conference on Computer Vision (ICCV), pages 1398-1406.
  8. Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531. · arXiv
  9. Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. 2017. Imitation learning: A sur- vey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35.
  10. Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, and By- ron C. Wallace. 2020. Learning to faithfully rational- ize by construction. In ACL.
  11. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015. Dbpedia-a large-scale, mul- tilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167-195.
  12. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
  13. Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. 2013. Rectifier nonlinearities improve neural net- work acoustic models. In in ICML Workshop on Deep Learning for Audio, Speech and Language Pro- cessing.
  14. Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukher- jee. 2021. Hatexplain: A benchmark dataset for ex- plainable hate speech detection. In AAAI.
  15. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In NeurIPS.
  16. Hosein Mohebbi, Ali Modarressi, and Moham- mad Taher Pilehvar. 2021. Exploring the role of BERT token representations to explain sentence prob- ing results. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 792-806, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  17. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. · arXiv
  18. Victor Sanh, Thomas Wolf, and Alexander Rush. 2020. Movement pruning: Adaptive sparsity by fine-tuning. Advances in Neural Information Processing Systems, 33:20378-20389.
  19. Roy Schwartz, Jesse Dodge, Noah Smith, and Oren Etzioni. 2020a. Green ai. Communications of the ACM, 63:54 -63.
  20. Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2020. Q-bert: Hessian based ultra low pre- cision quantization of bert. In AAAI.
  21. Karen Simonyan, Andrea Vedaldi, and Andrew Zis- serman. 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034. · arXiv
  22. D Smilkov, N Thorat, B Kim, F Viégas, and M Watten- berg. 2017. Smoothgrad: removing noise by adding noise. arxiv. arXiv preprint arxiv:1706.03825. · arXiv
  23. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
  24. Tianxiang Sun, Yunhua Zhou, Xiangyang Liu, Xinyu Zhang, Hao Jiang, Zhao Cao, Xuanjing Huang, and Xipeng Qiu. 2021. Early exiting with ensemble inter- nal classifiers. arXiv preprint arXiv:2105.13792. · arXiv
  25. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceed- ings of the 34th International Conference on Machine Learning-Volume 70, pages 3319-3328.
  26. Thierry Tambe, Coleman Hooper, Lillian Pentecost, Tianyu Jia, En-Yu Yang, Marco Donato, Victor Sanh, Paul N. Whatmough, Alexander M. Rush, David Brooks, and Gu-Yeon Wei. 2021. Edge- bert: Sentence-level energy optimizations for latency- aware multi-task nlp inference. MICRO-54: 54th Annual IEEE/ACM International Symposium on Mi- croarchitecture.
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30.
  28. Hanrui Wang, Zhekai Zhang, and Song Han. 2021. Spat- ten: Efficient sparse attention architecture with cas- cade token and head pruning. 2021 IEEE Interna- tional Symposium on High-Performance Computer Architecture (HPCA), pages 97-110.
  29. Hao Yuan, Yongjun Chen, Xia Hu, and Shuiwang Ji. 2019. Interpreting deep models for text analysis via optimization and regularization methods. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5717-5724.
  30. Omar Zaidan and Jason Eisner. 2008. Modeling an- notators: A generative approach to learning from annotator rationales. In Proceedings of the 2008 Conference on Empirical Methods in Natural Lan- guage Processing, pages 31-40, Honolulu, Hawaii. Association for Computational Linguistics.
  31. Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In NIPS.
  32. Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul Bennett, and Saurabh Tiwary. 2020. Transformer-xh: Multi-evidence reasoning with ex- tra hop attention. In International Conference on Learning Representations.
  33. Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020. Bert loses patience: Fast and robust inference with early exit. In Advances in Neural Information Processing Systems, volume 33, pages 18330-18341.

Cited by

Sorted by citations

Related

Authors
Modarressi, A., Mohebbi, H., & Pilehvar, M. T.
Affiliations
  • Iran University of Science and Technology (IR)
  • Khatam University (IR)
  • Tilburg University (NL)
ORCID
Venue
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2022
Pages
1-15
ISSN
0736-587X (ISSN-L)
DOI
10.18653/v1/2022.acl-long.1
OpenAlex
W4221138792
Cited by
18
References
0
Citations by year
2025: 6 · 2024: 6 · 2023: 4 · 2022: 2
FWCI
2.49
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • Research portal (Tilburg University) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NWA.1292.19.399)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
BERT, inference acceleration, token importance, resource-limited settings, text classification, rationales

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review