The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
Open accessAbstract
Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds of millions of parameters.However, these models suffer from heavy model sizes and high latency such that they cannot be deployed to resourcelimited mobile devices.In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model.Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to various downstream NLP tasks via simple fine-tuning.Basically, MobileBERT is a thin version of BERT LARGE , while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks.To train MobileBERT, we first train a specially designed teacher model, an invertedbottleneck incorporated BERT LARGE model.Then, we conduct knowledge transfer from this teacher to MobileBERT.Empirical studies show that MobileBERT is 4.3× smaller and 5.5× faster than BERT BASE while achieving competitive results on well-known benchmarks.On the natural language inference tasks of GLUE, MobileBERT achieves a GLUE score of 77.7 (0.6 lower than BERT BASE ), and 62 ms latency on a Pixel 4 phone.On the SQuAD v1.1/v2.0question answering task, MobileBERT achieves a dev F1 score of 90.0/79.2(1.5/2.1 higher than BERT BASE ).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“To train MobileBERT, first the authors trained a specially designed teacher model, and then, they transferred the knowledge from the teacher to MobileBERT, which is 4.3× smaller and 5.5× faster than the general BERT model.”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [57]
The cited source says what the claim attributes to it.
In the citing paper
“The distilled MobileBERT model is 4.3 × smaller than the BERT-base network and 5.5 × faster on a Pixel 4 mobile phone.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“MobileBERT [67] is a compressed version of the BERT-base model, and it runs 5.5 ×”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [67]
In the citing paper
“Moreover, models such as MobileBERT and SqueezeBERT employ bottleneck layers and grouped convolutions, leading to a 3× to 4× reduction in latency while maintaining >90% of the original model’s accuracy.”
From Tiny Language Models for Automation and Control: Overview, Potential Applications, and Future Research Directions (2025) · cites this paper as [68]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
(2010) · International Conference on Machine Learning · Cited by 13,995
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Research works printed but not linked: 2
- William B Dolan and Chris. Brockett. 2005. Auto- matically constructing a corpus of sentential para- phrases. In Proceedings of the International Work- shop on Paraphrasing. 7 https://tensorflow.org/lite/models/ bert_qa/overview
- Ganesh Jawahar, Benoît Sagot, Djamé Seddah, Samuel Unicomb, Gerardo Iñiguez, Márton Karsai, Yannick Léo, Márton Karsai, Carlos Sarraute, Éric Fleury, et al. 2019. What does bert learn about the structure of language? In 57th Annual Meeting of the Associa- tion for Computational Linguistics (ACL), Florence, Italy.
Cited by
Sorted by citations
Items 701–758 of 758
Parallax: Runtime Parallelization for Operator Fallbacks in Heterogeneous Edge Systems
Tang, C., Dai, H., & Chauhan, J. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2026.3695399
Ramali, C. A., Monica, S., Tedjasulaksana, J. J., Sagala, N. T. M., & Wulandhari, L. A. (2026) · International Conference on Current Research in Artificial Intelligence and Data Science (ICCRAIDS) · doi:10.1109/iccraids67816.2026.11519660
Exploring and improving knowledge distillation for pre-trained code models
Sun, W., Wu, R., Li, H., Fu, Y., Yu, M., & Yan, M. L. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10880-w
Lightweight Transformer Models for Resource–Constrained Environments: A Comparative Analysis
Kini, P., Sista, O., Prabhu, R., & Puthran, S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-24926-5_32
TinyLite: Efficient Edge Intelligence for Real-Time Spam Detection Under Device Constraints
Zhu, X., Polin, J. A., & Lu, S. (2026) · IEEE International Conference on Mobility, Operations, Services and Technologies (MOST) · doi:10.1109/most69733.2026.00042
Interfaze: The Future of AI is built on Task-Specific Small Models
Khurdula, H. V., Agarwal, V., & Khemlani, Y. D. (2026) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai68641.2026.11536205
Side-Channel Privacy Attacks on Energy-Efficient Mobile Agents: A Membership Inference Perspective
Ehsan, I., Niu, J., & Zhang, Y. (2026) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3812836.3814743
TrustSeed: Lightweight Attestation Protocol for Ensuring LLM Integrity
Alsharkawy, M., Ahmed, M. A., Nassar, H., Gonzalez, J., Khdr, H., Abboud, O., Xiao, X., & Henkel, J. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539273
Effective Lightweight Language Models for Edge Deployment in Low-Resource Settings
Panjolia, S., Chaudhary, A., Yadav, P., Uchoi, E., Singh, J., & Debbarma, K. (2026) · International Conference on Intelligent Systems in Engineering, Secured Systems and Cybersecurity (ICISESSC) · doi:10.1109/icisessc68634.2026.11542686
HILAL: Hessian-Informed Layer Allocation for Heterogeneous Analog–Digital Inference
Bessalah, A., Abdelmoumen, H., Benatchba, K., & Benmeziane, H. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539316
Zhang, N., Mao, H., Zhang, S., Liu, X. M., & Jiang, H. (2026) · Neural Processing Letters · doi:10.1007/s11063-026-11857-y
FaStrA: Fast and Efficient Network Structure-Aware Layer Fusion on Deep Learning Accelerators
Choi, J., Joo, W., Gafiyatullin, A., Karpukhin, S., Kwon, K., & Kim, M. (2026) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3820047
RELTrack: A language-guided RGB-event object tracking framework
Alansari, S., Al‐Sumaiti, A. S., Hosani, K. H. A., & Hussain, S. F. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104975
Comparison of Transformer-based Deep Learning Methods for the Paraphrase Identification Task
Vrublevskyi, V., & Marchenko, O. O. (2022) · International Scientific Conference Information Technology and Implementation (IT&I) · doi:10.55056/ceur-ws.org/vol-3624/short_5.pdf
Martínez-Cruz, R., López-López, Á. J., Portela, J., & Zayat, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7247158
Witthuhn, N., Sibiya, M., & Sumbwanyambe, M. (2026) · Education Sciences · doi:10.3390/educsci16071048 · Cited by 1
Tiny Models, Tough Limits: Benchmarking and Optimizing Small Language Models for Edge Deployment
Makki, A. P., Lago, L. A., Saleem, C., ElShabasy, M., & Pissinou, N. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-22500-9_1
Kim, H., Lee, J., Choi, M., Lee, J., & Cho, J. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-22500-9_24
Federated DistilBERT: Privacy-Preserving Small Language Models for Question Answering
Manantan, L., Saeed, N., Fernando, J., Sharma, M., Arora, P., Gollapudi, S. D. R., Braytee, A., Huang, W., Hua, J., & Anaissi, A. (2026) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-032-23260-1_17
Wue, Y., Ye, L., Zhang, H., Xie, R., & Peng, W. (2026) · Global Information Infrastructure and Networking Symposium · doi:10.1109/giis69881.2026.11585752
Collaborative Inference Scheme with Small and Large Language Models in Edge Networks
Xu, X., Feng, G., Liu, Y., Qin, S., & Wang, Y. (2025) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/gcwkshps68340.2025.11591146
Fake News Detection in Tagalog: Benchmarking and Ensemble of Compact Transformer Models
Cuñado, J. T., Toledo, J. V. V., & Bandalan, C. D. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-20897-2_30
Hierarchical Knowledge Distillation with Meta Learning
Gogoi, M., Bhargava, A., & Verma, S. (2026) · International Conference on Computational Intelligence and Communication Networks (CICN) · doi:10.1109/cicn70047.2026.11594229
Possible Distillation for Abstractive Summarization: Reducing Hallucination in Compact LLMs
Abishethvarman, V., Balasubramaniam, G., Prasanth, S., Kumara, B. T. G. S., & Kuhaneswaran, B. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-0137-2_15
Tian, M., Zhao, Q., He, Y. L., & Wang, D. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3432-5_40
Mobile Webpage Phishing Detection through Model Distillation and Stacking
Re, G. L., Morana, M., & Rizzo, G. (2026) · International Conference on Communications · doi:10.1109/icc59461.2026.11588184
MEMBERT: Inductive-Bias-Driven Modeling of Depression Emotions
Chen, Z., & He, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2759-4_21
Yuan, H., Bi, J., Zhou, M., & Buyya, R. (2026) · Internet of things · doi:10.1007/978-3-032-18400-9_2
Pilli, R. M. R., Koch, J., & Wright, C. (2026) · IEEE International Conference on AI and Data Analytics (ICAD) · doi:10.1109/icad69378.2026.11609131
Patel, M., Sharma, Y., K., P. R., Subramani, C., B, V., & Palanikumar, S. (2026) · 2026 5th International Conference on Computer Networks, Big Data and IoT (ICCBI) · doi:10.1109/iccbi68589.2026.11619784
Distilling Sentiment: Leveraging High-Resource Models for Low-Resource Languages
Mallya, N. N., Ramasamy, P., Pakhare, S., & Hegde, S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-31216-7_47
Cheng, Z., Lai, L., Liu, Y., & Sun, Y. (2026) · Research Square · doi:10.2139/ssrn.6698538
An Offline AI-Driven Survey Automation Framework Using TinyBERT and Progressive Web Apps
Patil, D. D., Deshmukh, P., Waghmare, R., Mukade, A., & Surye, S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-27154-9_11
LiteLLM: a lightweight transformer architecture for efficient short-text classification
Al-Kaabi, H., Al-Ibraheemi, F. A., & Jasim, A. K. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12379-3
TWD-KD: Temperature-Guided Wavelet Decomposition Knowledge Distillation
Bai, Z., & Lei, X. (2026) · International Conference on Computer Vision, Image and Deep Learning (CVIDL) · doi:10.1109/cvidl70130.2026.11637685
Tanjaya, R. D., Ecklesia, G., Lucky, H., & Pradana, R. C. (2026) · doi:10.1109/icoict71118.2026.11664832
Location-Aware Language Models via Secondary Embeddings
Srinivasagan, G., & Georges, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37249-9_14
Sani, M. S., Iranmanesh, S., & Raad, R. (2026) · Sensors · doi:10.3390/s26185862
Benchmarking Deep Learning Models for Enhancing Stock Price Prediction with Sentiment Fusion
Ma, R., Wang, H., Xie, D., & Alve, J. C. (2026) · International Conference on Image, Vision and Computing (ICIVC) · doi:10.1109/icivc70586.2026.11686738
Aggarwal, S., & Vishwakarma, D. K. (2026) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.70958
Esmaeili, H., KAZEMI, M. A. A., Radfar, R., & Pilevari, N. (2026) · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · doi:10.55730/1300-0632.4275
Vigil NLP: Intelligent Threat Detection and Alerting System
Sri, N. T., Kumar, A. R., Kumar, G. S. U., Pavana, V., Dharmesh, R., & Sandeep, D. S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-30957-0_37
Hu, Z., Peng, C., Peng, Z., Cai, Z., Zhang, Z., Sun, Z., Hu, J., & Shouyu, L. (2026) · Mathematics · doi:10.3390/math14193479
Self-citationL-VLA:Efficient Multimodal Decision-Making Framework for Integrated Perception-Action
Arora, M. (2026) · IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT) · doi:10.1109/conecct68978.2026.11700742
A comprehensive survey on multimodal sentiment analysis: Approaches, challenges and open issues
Hemmatian, F., Sabbaghi, A., & Sohrabi, M. K. (2026) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2026.103138
Distilling Encoder-Decoder and Decoder-Only Models into BERT for NLP Tasks
Nathanael, K., Danielson, Zhang, J. A., Hidayaturrahman, H., & Santika, S. P. (2026) · International Conference on ICT for Smart Society (ICISS) · doi:10.1109/iciss71466.2026.11715411
Khatoon, K. S., Naiyeer, N. M., Reshma, P., Sulthana, S. N., Rajeshwari, C., & Harish, O. (2026) · International Conference on Soft Computing for Security Applications (ICSCSA) · doi:10.1109/icscsa69367.2026.11707906
Gaze-Based Attention-Driven Multimodal Single-Object Inference for Microcontroller-Class Edge Vision
Glorioso, S. M., Arena, A. P., Guarneri, I., & Sorce, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39379-1_45
Depth Adaptive Efficient Visual Autoregressive Modeling
Li, C., Cao, T., & Zhao, S. (2026) · doi:10.1109/cvprf71909.2026.00407
RegAct: A Registry-Constrained Text-to-Action Framework with Closed-Loop Execution for Edge Robots
Zhong, Z., Xu, W., & Ma, Y. (2026) · WRC Symposium on Advanced Robotics and Automation (WRC SARA) · doi:10.1109/wrcsara71458.2026.11732547
Qin, Y., Lyu, J., Li, Z., & Cao, P. (2026) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3770743.3804085
Enhancing monocular 3D semantic scene graph prediction via cross-modality knowledge distillation
Bae, J., Lee, M., Buu, S.-J., Choi, S.-M., & Lee, S. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.115175
Lightweight Language Models for Resource-Constrained Environments
Mohammad, R. J., Roy, T., Singh, B. P., Tekale, K. M., Manchanda, S., & Sadat, Q. T. (2026) · doi:10.1109/incect72370.2026.11742839
Periodic Segment-Wise Knowledge Distillation with Adaptive Temperature
Li, G. (2026) · Journal of Computer and Communications · doi:10.4236/jcc.2026.1410005
Concept Drifts, Detector Evolves: Malware Detection Made Easy with LLMalware
Ma, Z., Shen, L., Huang, X., Zhou, K., & Zheng, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-42936-0_1
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
Dong, Y., Cordonnier, J.-B., & Loukas, A. (2021) · arXiv · doi:10.48550/arxiv.2103.03404 · Cited by 195
I-BERT: Integer-only BERT Quantization
Kim, S., Gholami, A., Yao, Z., Mahoney, M. W., & Keutzer, K. (2021) · arXiv · doi:10.48550/arxiv.2101.01321 · Cited by 186
NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM
Holmes, C., Zhang, M., He, Y., & Wu, B. (2021) · arXiv · doi:10.48550/arxiv.2110.15766 · Cited by 7
Metadata
- Authors
- Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., & Zhou, D.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 2158-2170
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3034457371
- Cited by
- 757
- References
- 54
- Citations by year
- 2026: 94 · 2025: 134 · 2024: 98 · 2023: 128 · 2022: 119 · 2021: 140 · 2020: 43
- FWCI
- 41.07
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- MobileBERT, BERT, model compression, knowledge distillation, teacher-student learning, self-attention, feedforward neural networks, mobile devices, resource-constrained deployment, natural language inference, question answering, adhesives, SQuAD
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
