The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
Open accessAbstract
Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds of millions of parameters.However, these models suffer from heavy model sizes and high latency such that they cannot be deployed to resourcelimited mobile devices.In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model.Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to various downstream NLP tasks via simple fine-tuning.Basically, MobileBERT is a thin version of BERT LARGE , while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks.To train MobileBERT, we first train a specially designed teacher model, an invertedbottleneck incorporated BERT LARGE model.Then, we conduct knowledge transfer from this teacher to MobileBERT.Empirical studies show that MobileBERT is 4.3× smaller and 5.5× faster than BERT BASE while achieving competitive results on well-known benchmarks.On the natural language inference tasks of GLUE, MobileBERT achieves a GLUE score of 77.7 (0.6 lower than BERT BASE ), and 62 ms latency on a Pixel 4 phone.On the SQuAD v1.1/v2.0question answering task, MobileBERT achieves a dev F1 score of 90.0/79.2(1.5/2.1 higher than BERT BASE ).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“To train MobileBERT, first the authors trained a specially designed teacher model, and then, they transferred the knowledge from the teacher to MobileBERT, which is 4.3× smaller and 5.5× faster than the general BERT model.”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [57]
The cited source says what the claim attributes to it.
In the citing paper
“The distilled MobileBERT model is 4.3 × smaller than the BERT-base network and 5.5 × faster on a Pixel 4 mobile phone.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“MobileBERT [67] is a compressed version of the BERT-base model, and it runs 5.5 ×”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [67]
In the citing paper
“Moreover, models such as MobileBERT and SqueezeBERT employ bottleneck layers and grouped convolutions, leading to a 3× to 4× reduction in latency while maintaining >90% of the original model’s accuracy.”
From Tiny Language Models for Automation and Control: Overview, Potential Applications, and Future Research Directions (2025) · cites this paper as [68]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
(2010) · International Conference on Machine Learning · Cited by 13,995
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Research works printed but not linked: 2
- William B Dolan and Chris. Brockett. 2005. Auto- matically constructing a corpus of sentential para- phrases. In Proceedings of the International Work- shop on Paraphrasing. 7 https://tensorflow.org/lite/models/ bert_qa/overview
- Ganesh Jawahar, Benoît Sagot, Djamé Seddah, Samuel Unicomb, Gerardo Iñiguez, Márton Karsai, Yannick Léo, Márton Karsai, Carlos Sarraute, Éric Fleury, et al. 2019. What does bert learn about the structure of language? In 57th Annual Meeting of the Associa- tion for Computational Linguistics (ACL), Florence, Italy.
Cited by
Sorted by citations
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
A Metaverse: Taxonomy, Components, Applications, and Open Challenges
Park, S., & Kim, Y. (2022) · IEEE Access · doi:10.1109/access.2021.3140175 · Cited by 2,046
TinyBERT: Distilling BERT for natural language understanding
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.372 · Cited by 1,786
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development
Wu, T., He, S., Liu, J., Sun, S., Liu, K., Han, Q.-L., & Tang, Y. (2023) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123618 · Cited by 1,452
Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
Menghani, G. (2023) · ACM Computing Surveys · doi:10.1145/3578938 · Cited by 732
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services
Xu, M., Du, H., Niyato, D. T., Kang, J., Xiong, Z., Mao, S., Han, Z., Jamalipour, A., Kim, D. I., Shen, X., et al. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3353265 · Cited by 518
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning
Desislavov, R., Martínez-Plumed, F., & Hernández-Orallo, J. (2023) · Sustainable Computing: Informatics and Systems · doi:10.1016/j.suscom.2023.100857 · Cited by 269
“, MobileBERT [Sun et al., 2020] and SqueezeBERT [Iandola et al.”
BackgroundBERT applications in natural language processing: a review
Gardazi, N. M., Daud, A. Y., Malik, M. K., Bukhari, A. A., Alsahfi, T., & Alshemaimri, B. K. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11162-5 · Cited by 354
Parameter-Efficient Transfer Learning with Diff Pruning
Guo, D., Rush, A., & Kim, Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.378 · Cited by 220
The effect of choosing optimizer algorithms to improve computer vision tasks: a comparative study
Hassan, E., Shams, M. Y., Hikal, N. A., & Elmougy, S. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13820-0 · Cited by 265
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
Ganesh, P., Chen, Y., Lou, X., Khan, M. A., Yang, Y., Sajjad, H., Nakov, P., Chen, D., & Winslett, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00413 · Cited by 190
Self-citationAdapterDrop: On the Efficiency of Adapters in Transformers
Rücklé, A., Geigle, G., Glockner, M., Beck, T., Pfeiffer, J., Reimers, N., & Gurevych, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.626 · Cited by 271
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer
Vu, T., Lester, B., Constant, N., Al‐Rfou, R., & Cer, D. M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.346 · Cited by 267
Kheddar, H. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103347 · Cited by 240
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language ProcessingPreprint
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.05542 · Cited by 357
Lewis, P., Ott, M., Du, J., & Stoyanov, V. (2020) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2020.clinicalnlp-1.17 · Cited by 196
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
Xu, C., Zhou, W., Ge, T., Wei, F., & Zhou, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.633 · Cited by 213
A Review on Edge Large Language Models: Design, Execution, and Applications
Zheng, Y., Chen, Y., Qian, B., Shi, X., Shu, Y., & Chen, J. (2025) · ACM Computing Surveys · doi:10.1145/3719664 · Cited by 169
Bringing ai to edge: from deep learning’s perspective
Liu, D., Kong, H., Luo, X., Liu, W., & Subramaniam, R. (2022) · Neurocomputing · doi:10.1016/j.neucom.2021.04.141 · Cited by 171
End-to-End Transformer-Based Models in Textual-Based NLP
Rahali, A., & Akhloufi, M. A. (2023) · AI · doi:10.3390/ai4010004 · Cited by 209
The NLP Cookbook: Modern Recipes for Transformer Based Deep Learning Architectures
Singh, S. K., & Mahmood, A. (2021) · IEEE Access · doi:10.1109/access.2021.3077350 · Cited by 178
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154
Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI
Yao, J., Zhang, S., Yao, Y., Wang, F., Ma, J., Zhang, J., Chu, Y., Ji, L., Jia, K., Shen, T., et al. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3178211 · Cited by 161
Keyword Transformer: A Self-Attention Model for Keyword Spotting
Berg, A., O’Connor, M., & Cruz, M. T. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1286 · Cited by 179
Learned Token Pruning for Transformers
Kim, S., Shen, S., Thorsley, D., Gholami, A., Kwon, W., Hassoun, J., & Keutzer, K. (2022) · Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3534678.3539260 · Cited by 139
Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language ProcessingPreprint
Dai, Z., Lai, G., Yang, Y., & Le, Q. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03236 · Cited by 159
Self-citationGOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference
Zadeh, A. H., Edo, I., Awad, O. M., & Moshovos, A. (2020) · 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00071 · Cited by 10
Structured Pruning Learns Compact and Accurate Models
Xia, M., Zhong, Z., & Chen, D. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.107 · Cited by 136
ConvBERT: Improving BERT with Span-based Dynamic ConvolutionPreprint
Jiang, Z., Yu, W., Zhou, D., Chen, Y., Feng, J., & Yan, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.02496 · Cited by 160
Self-citationMiniViT: Compressing Vision Transformers with Weight Multiplexing
Zhang, J., Peng, H., Wu, K., Liu, M., Xiao, B., Fu, J., & Yuan, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01183 · Cited by 135
JAKET: Joint Pre-training of Knowledge Graph and Language Understanding
Yu, D., Zhu, C., Yang, Y., & Zeng, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21417 · Cited by 144
Self-citationBlock Pruning For Faster Transformers
Lagunas, F., Charlaix, E., Sanh, V., & Rush, A. M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.829 · Cited by 204
EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference
Tambe, T., Hooper, C. R. C., Pentecost, L., Jia, T., Yang, E.-Y., Donato, M., Sanh, V., Whatmough, P. N., Rush, A. M., Brooks, D. M., et al. (2021) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3466752.3480095 · Cited by 145
ALP-KD: Attention-Based Layer Projection for Knowledge Distillation
Passban, P., Wu, Y., Rezagholizadeh, M., & Liu, Q. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17610 · Cited by 127
Neural Architecture Search for Transformers: A Survey
Chitty-Venkata, K. T., Emani, M., Vishwanath, V., & Somani, A. K. (2022) · IEEE Access · doi:10.1109/access.2022.3212767 · Cited by 118
SqueezeBERT: What can computer vision teach NLP about efficient neural networks?
Iandola, F., Shaw, A., Krishna, R., & Keutzer, K. (2020) · Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing · doi:10.18653/v1/2020.sustainlp-1.17 · Cited by 114
ZeroGen: Efficient Zero-shot Learning via Dataset Generation
Ye, J., Gao, J., Li, Q., Xu, H., Feng, J., Wu, Z., Yu, C., & Kong, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.801 · Cited by 200
Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better
Zi, B., Zhao, S., Ma, X., & Jiang, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01613 · Cited by 128
Efficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 156
Compression of Deep Learning Models for Text: A Survey
Gupta, M., & Agrawal, P. (2022) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3487045 · Cited by 128
A Review on Deep-Learning-Based Cyberbullying Detection
Hasan, M. T., Hossain, M. A. E., Mukta, M. S. H., Akter, A., Ahmed, M., & Islam, S. (2023) · Future Internet · doi:10.3390/fi15050179 · Cited by 103
“MobileBERT [57]: BERT suffers from heavy model sizes and high latency, which cannot be applied to limited resource devices such as mobile phones.”
BackgroundSEED: Self-supervised Distillation For Visual RepresentationPreprint
Fang, Z., Wang, J., Wang, L., Zhang, L., Yang, Y., & Liu, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.04731 · Cited by 134
Self-citationFrom Data to Discovery: Recent Trends of Machine Learning in Metal–Organic Frameworks
Park, J., Kim, H., Kang, Y., Lim, Y., & Kim, J. (2024) · JACS Au · doi:10.1021/jacsau.4c00618 · Cited by 102
From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough
Mars, M. (2022) · Applied Sciences · doi:10.3390/app12178805 · Cited by 91
Understanding and Overcoming the Challenges of Efficient Transformer Quantization
Bondarenko, Y., Nagel, M., & Blankevoort, T. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.627 · Cited by 172
PIMNet: A Parallel, Iterative and Mimicking Network for Scene Text Recognition
Qiao, Z. Q., Zhou, Y., Jin, W., Wang, W., Zhang, Y., Jiang, N., Wang, H., & Wang, W. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475238 · Cited by 87
AutoScale: Energy Efficiency Optimization for Stochastic Edge Inference Using Reinforcement Learning
Kim, Y. G., & Wu, C.-J. (2020) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00090 · Cited by 88
Wang, X., Wang, B., Wu, Y., Ning, Z., Guo, S., & Yu, F. R. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3446585 · Cited by 101
Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
Yang, C., Zhu, Y. T., Wang, L., Wang, Y., Chen, Q., Gao, C., Yan, B., & Chen, Y. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3699518 · Cited by 128
Rethinking embedding coupling in pre-trained language modelsPreprint
Chung, H. W., Févry, T., Tsai, H., Johnson, M., & Ruder, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12821 · Cited by 125
Teng, T. H., & Varathan, K. D. (2023) · IEEE Access · doi:10.1109/access.2023.3275130 · Cited by 77
A Survey of Robot Intelligence with Large Language Models
Jeong, H., Lee, H., Kim, C., & Shin, S. (2024) · Applied Sciences · doi:10.3390/app14198868 · Cited by 89
Annealing Knowledge Distillation
Jafari, A., Rezagholizadeh, M., Sharma, P., & Ghodsi, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.212 · Cited by 79
Compressing Visual-linguistic Model via Knowledge Distillation
Fang, Z., Wang, J., Hu, X., Wang, L., Yang, Y., & Liu, Z. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00146 · Cited by 81
Self-citationDeLighT: Deep and Light-weight TransformerPreprint
Mehta, S., Ghazvininejad, M., Iyer, S. V., Zettlemoyer, L., & Hajishirzi, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.00623 · Cited by 113
Structured Pruning of a BERT-based Question Answering ModelPreprint
McCarley, J. S., Chakravarti, R., & Sil, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.06360 · Cited by 139
Efficient Federated Learning for Modern NLP
Cai, D., Wu, Y., Wang, S., Lin, F. X., & Xu, M. (2023) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3570361.3592505 · Cited by 83
Transformer technology in molecular science
Jian, J., Ke, L., Chen, L., Dou, B., Zhu, Y., Liu, J., Zhang, B., Zhou, T., & Wei, G. (2024) · Wiley Interdisciplinary Reviews Computational Molecular Science · doi:10.1002/wcms.1725 · Cited by 73
DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering
Cao, Q., Trivedi, H. K., Balasubramanian, A., & Balasubramanian, N. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.411 · Cited by 73
Exploring the Impact of AI on The EFL Context: A Case Study of Saudi Universities
Alhalangy, A., & AbdAlgane, M. (2023) · Journal of Intercultural Communication · doi:10.36923/jicc.v23i2.125 · Cited by 118
Can a Student Large Language Model Perform as Well as Its Teacher?
Gholami, S., & Omar, M. (2024) · Advances in medical technologies and clinical practice book series · doi:10.4018/979-8-3693-1906-2.ch007 · Cited by 65
Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation
Xu, Y., Qiu, X., Zhou, L.-G., & Huang, X. (2023) · Journal of Computer Science and Technology · doi:10.1007/s11390-021-1119-0 · Cited by 89
DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devices
Xu, G., Hao, Z., Luo, Y., Hu, H., An, J., & Mao, S. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3315138 · Cited by 68
Contrastive Distillation on Intermediate Representations for Language Model Compression
Sun, S., Gan, Z., Fang, Y., Cheng, Y., Wang, S., & Liu, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.36 · Cited by 77
A Survey on Model Compression and Acceleration for Pretrained Language Models
Xu, C., & McAuley, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26255 · Cited by 85
Pre-trained Summarization DistillationPreprint
Shleifer, S., & Rush, A. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13002 · Cited by 121
Lamaakal, I., Maleh, Y., El Makkaoui, K., Ouahbi, I., Pławiak, P., Alfarraj, O., Almousa, M., & Abd El-Latif, A. A. (2025) · Sensors · doi:10.3390/s25051318 · Cited by 60
Oliaee, A. H., Das, S., Liu, J., & Rahman, M. A. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100007 · Cited by 65
Memory-efficient DNN training on mobile devices
Gim, I., & Ko, J. (2022) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3498361.3539765 · Cited by 61
Fu, H., Zhou, S., Yang, Q., Tang, J., Liu, G., Liu, K., & Li, X. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17518 · Cited by 58
Two Stage Transformer Model for COVID-19 Fake News Detection and Fact CheckingPreprint
Vijjali, R., Potluri, P., Kumar, S., & Teki, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13253 · Cited by 93
Survey of BERT (Bidirectional Encoder Representation Transformer) types
Mohammed, A. H., & Ali, A. H. (2021) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1963/1/012173 · Cited by 62
Piccolo: Exposing Complex Backdoors in NLP Transformer Models
Liu, Y., Shen, G., Tao, G., An, S., Ma, S., & Zhang, X. (2022) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46214.2022.9833579 · Cited by 65
Xu, J., Tan, X., Luo, R., Song, K., Li, J., Qin, T., & Liu, T. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467262 · Cited by 63
Jin, Z., Lalwani, A., Vaidhya, T., Shen, X., Ding, Y., Lyu, Z., Sachan, M., Mihalcea, R., & Schölkopf, B. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.532 · Cited by 52
Guo, Z., Tang, F., Luo, L., Zhao, M., & Kato, N. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3568637 · Cited by 52
LLM-Based Multi-Agent Decision-Making: Challenges and Future Directions
Sun, C., Huang, S., & Pompili, D. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3562371 · Cited by 65
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts
Zhang, Z., Lin, Y., Liu, Z., Li, P., Sun, M., & Zhou, J. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.71 · Cited by 48
BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover’s Distance
Li, J., Liu, X., Zhao, H., Xu, R., Yang, M., & Jin, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.242 · Cited by 48
Compression of Generative Pre-trained Language Models via Quantization
Tao, C., Hou, L., Zhang, W., Shang, L., Jiang, X., Liu, Q., Luo, P., & Wong, N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.331 · Cited by 52
Compressing Pre-trained Models of Code into 3 MB
Shi, J., Yang, Z., Xu, B., Kang, H. J., & Lo, D. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556964 · Cited by 47
Which *BERT? A Survey Organizing Contextualized Encoders
Xia, P., Wu, S., & Durme, B. V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.608 · Cited by 57
TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference
Ye, D., Lin, Y., Huang, Y., & Sun, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.463 · Cited by 68
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
Chen, X., Cheng, Y., Wang, S., Gan, Z., Wang, Z., & Liu, J. (2021) · doi:10.18653/v1/2021.acl-long.171 · Cited by 86
Cui, B., Li, Y., & Zhang, Z. M. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.05.084 · Cited by 47
Visual language navigation: a survey and open challenges
Park, S., & Kim, Y. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10174-9 · Cited by 49
Galke, L. P. A., & Scherp, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.279 · Cited by 57
Federated and edge learning for large language models
Piccialli, F., Chiaro, D., Qi, P., Bellandi, V., & Damiani, E. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102840 · Cited by 56
Meta Learning for Natural Language Processing: A Survey
Lee, H.-Y., Li, S.-W. D., & Vu, T. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.49 · Cited by 67
Dynamic Contrastive Distillation for Image-Text Retrieval
Rao, J., Ding, L., Qi, S., Fang, M., Liu, Y., Shen, L., & Tao, D. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3236837 · Cited by 46
MobileDiffusion: Instant Text-to-Image Generation on Mobile Devices
Zhao, Y., Xu, Y., Xiao, Z., Jia, H., & Hou, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73033-7_13 · Cited by 94
Geffen, Y., Ofran, Y., & Unger, R. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac474 · Cited by 40
FPT: Fine-Grained Detection of Driver Distraction Based on the Feature Pyramid Vision Transformer
Wang, H., Chen, J., Huang, Z., Li, B., Lv, J., Xi, J., Wu, B., Zhang, J., & Wu, Z. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3219676 · Cited by 48
Exploring Privileged Features for Relation Extraction with Contrastive Student-Teacher Learning
Xiaoyan, Z., Yang, M., Qu, Q., Xu, R., & Li, J. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3161584 · Cited by 39
Lightweight transformers for clinical natural language processing
Rohanian, O., Nouriborji, M., Jauncey, H., Kouchaki, S., Nooralahzadeh, F., Clifton, L., Merson, L., & Clifton, D. A. (2024) · Natural Language Engineering · doi:10.1017/s1351324923000542 · Cited by 51
On the importance of pre-training data volume for compact language models
Micheli, V., d’Hoffschmidt, M., & Fleuret, F. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.632 · Cited by 40
Yuan, M., Zhang, L., He, F., Tong, X., & Li, X. (2022) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3495243.3517016 · Cited by 40
Liu, Y., Ju, S., & Wang, J. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02481-8 · Cited by 40
Metadata
- Authors
- Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., & Zhou, D.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 2158-2170
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3034457371
- Cited by
- 757
- References
- 54
- Citations by year
- 2026: 94 · 2025: 134 · 2024: 98 · 2023: 128 · 2022: 119 · 2021: 140 · 2020: 43
- FWCI
- 41.07
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- MobileBERT, BERT, model compression, knowledge distillation, teacher-student learning, self-attention, feedforward neural networks, mobile devices, resource-constrained deployment, natural language inference, question answering, adhesives, SQuAD
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
