BERT Loses Patience: Fast and Robust Inference with Early Exit
Open accessThe DOI matches its OpenAlex record; 30 of the 30 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01083 · Cited by 9,982
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00716 · Cited by 9,982
Cited by
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
“Early Exit DeeBERT [110], RightTool [111], FastBERT [112], PABEE [113], Liao et al.”
BackgroundMethodLin, T., Wang, Y., Liu, X., & Qiu, X. (2021) · arXiv · doi:10.48550/arxiv.2106.04554 · Cited by 1,420
Lin, T., Wang, Y., Liu, X., & Qiu, X. (2022) · AI Open · doi:10.1016/j.aiopen.2022.10.001 · Cited by 1,422
Dynamic Neural Networks: A Survey
Han, Y., Huang, G., Song, S., Yang, L., Wang, H., & Wang, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3117837 · Cited by 848
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges
Matsubara, Y., Levorato, M., & Restuccia, F. (2023) · ACM Computing Surveys · doi:10.1145/3527155 · Cited by 304
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
Ganesh, P., Chen, Y., Lou, X., Khan, M. A., Yang, Y., Sajjad, H., Nakov, P., Chen, D., & Winslett, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00413 · Cited by 190
A Review on Edge Large Language Models: Design, Execution, and Applications
Zheng, Y., Chen, Y., Qian, B., Shi, X., Shu, Y., & Chen, J. (2025) · ACM Computing Surveys · doi:10.1145/3719664 · Cited by 169
DynaBERT: Dynamic BERT with Adaptive Width and Depth
Hou, L., Huang, Z., Shang, L., Jiang, X., Chen, X., & Liu, Q. (2020) · arXiv · doi:10.48550/arxiv.2004.04037 · Cited by 143
“In [4, 7, 6, 13, 29, 34], Transformerbased models with adaptive depth are proposed to dynamically select some of the Transformer layers during inference.”
BackgroundWidth & Depth Pruning for Vision Transformers
Yu, F., Huang, K., Wang, M., Cheng, Y., Chu, W., & Cui, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20222 · Cited by 95
The Art of Abstention: Selective Prediction and Error Regularization for Natural Language Processing
Xin, J., Tang, R., Yu, Y., & Lin, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.84 · Cited by 45
BinaryBERT: Pushing the Limit of BERT Quantization
Bai, H., Zhang, W., Hou, L., Shang, L., Jin, J., Jiang, X., Liu, Q., Lyu, M., & King, I. (2020) · arXiv · doi:10.48550/arxiv.2012.15701 · Cited by 42
You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model
Tang, S., Wang, Y., Kong, Z., Zhang, T., Li, Y., Ding, C., Wang, Y., Liang, Y., & Xu, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01038 · Cited by 37
“Since the experiments of DeeBERT [51] and PAEBB [60] were originally conducted on BERT [4], we implement it in decoder part on OFA architecture.”
BackgroundMethodModel Tells You What to Discard: Adaptive KV Cache Compression for LLMs
Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2310.01801 · Cited by 13
AutoMix: Automatically Mixing Language Models
Aggarwal, P., Madaan, A., Anand, A., Potharaju, S. P., Mishra, S., Zhou, P., Gupta, A., Rajagopal, D., Kappaganthu, K., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.12963 · Cited by 19
“…sequentially queries models of disparate sizes and capabilities, at each step determining whether to accept the current output or route to a more capable, but computationally intensive, model (Liu et al., 2020; Zhou et al., 2020; Madaan and Yang, 2022; Geng et al., 2021; Schuster et al., 2022).”
BackgroundFINCH: Prompt-guided Key-Value Cache Compression for Large Language Models
Corallo, G., & Papotti, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00716 · Cited by 5
“Proposed solutions also involve learning to skip layers in the transformer architecture (Guan et al., 2022; Zhou et al., 2020) or to select the most critical tokens for performance (Huang et al., 2022).”
Metadata
- Authors
- Zhou, W., Xu, C., Ge, T., McAuley, J., Xu, K., & Wei, F.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2020-06-07
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.04152
- OpenAlex
- W3033737024
- Cited by
- 44
- References
- 30
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.