The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
Open accessAbstract
Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds of millions of parameters.However, these models suffer from heavy model sizes and high latency such that they cannot be deployed to resourcelimited mobile devices.In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model.Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to various downstream NLP tasks via simple fine-tuning.Basically, MobileBERT is a thin version of BERT LARGE , while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks.To train MobileBERT, we first train a specially designed teacher model, an invertedbottleneck incorporated BERT LARGE model.Then, we conduct knowledge transfer from this teacher to MobileBERT.Empirical studies show that MobileBERT is 4.3× smaller and 5.5× faster than BERT BASE while achieving competitive results on well-known benchmarks.On the natural language inference tasks of GLUE, MobileBERT achieves a GLUE score of 77.7 (0.6 lower than BERT BASE ), and 62 ms latency on a Pixel 4 phone.On the SQuAD v1.1/v2.0question answering task, MobileBERT achieves a dev F1 score of 90.0/79.2(1.5/2.1 higher than BERT BASE ).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“To train MobileBERT, first the authors trained a specially designed teacher model, and then, they transferred the knowledge from the teacher to MobileBERT, which is 4.3× smaller and 5.5× faster than the general BERT model.”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [57]
The cited source says what the claim attributes to it.
In the citing paper
“The distilled MobileBERT model is 4.3 × smaller than the BERT-base network and 5.5 × faster on a Pixel 4 mobile phone.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“MobileBERT [67] is a compressed version of the BERT-base model, and it runs 5.5 ×”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [67]
In the citing paper
“Moreover, models such as MobileBERT and SqueezeBERT employ bottleneck layers and grouped convolutions, leading to a 3× to 4× reduction in latency while maintaining >90% of the original model’s accuracy.”
From Tiny Language Models for Automation and Control: Overview, Potential Applications, and Future Research Directions (2025) · cites this paper as [68]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
(2010) · International Conference on Machine Learning · Cited by 13,995
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Research works printed but not linked: 2
- William B Dolan and Chris. Brockett. 2005. Auto- matically constructing a corpus of sentential para- phrases. In Proceedings of the International Work- shop on Paraphrasing. 7 https://tensorflow.org/lite/models/ bert_qa/overview
- Ganesh Jawahar, Benoît Sagot, Djamé Seddah, Samuel Unicomb, Gerardo Iñiguez, Márton Karsai, Yannick Léo, Márton Karsai, Carlos Sarraute, Éric Fleury, et al. 2019. What does bert learn about the structure of language? In 57th Annual Meeting of the Associa- tion for Computational Linguistics (ACL), Florence, Italy.
Cited by
Sorted by citations
Items 501–600 of 758
Lv, Y., Wang, L., Ge, M., Li, S., Ding, C., Pan, L., Wang, Y. G., Dang, J., & Honda, K. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746113 · Cited by 2
Learning Adaptive Axis Attentions in Fine-tuning: Beyond Fixed Sparse Attention Patterns
Wang, Z., Gu, J., Kuen, J., Zhao, H., Morariu, V. I., Zhang, R., Nenkova, A., Sun, T. S., & Shang, J. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.74 · Cited by 3
Falaki, A. A., & Gras, R. (2022) · International Conference on Machine Learning and Applications · doi:10.1109/icmla55696.2022.00030 · Cited by 2
Udagawa, T., Trivedi, A., Merler, M., & Bhattacharjee, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.3 · Cited by 3
Multi-word Tokenization for Sequence Compression
Gee, L., Rigutini, L., Ernandes, M., & Zugarini, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.58 · Cited by 9
Layer Compression of Deep Networks with Straight Flows
Gong, C., Du, X., Bhushanam, B., Wu, L., Liu, X., Choudhary, D., Kejariwal, A., & Liu, Q. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i11.29107 · Cited by 2
Architectures for Machine Learning
Yang, Y., Chen, C., & Wang, Z. (2024) · Springer eBooks · doi:10.1007/978-981-97-9314-3_12 · Cited by 2
Self-citationEmpLite: A Lightweight Sequence Labeling Model for Emphasis Selection of Short TextsPreprint
Agarwal, V., Ghosh, S. K., Chalamalasetti, K., Challa, B., Kumari, S., Harshavardhana, H., & Raja, B. R. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.03025 · Cited by 2
How Good Is NLP? A Sober Look at NLP Tasks through the Lens of Social ImpactPreprint
Jin, Z., Chauhan, G., Tse, B., Sachan, M., & Mihalcea, R. F. (2021) · arXiv (Cornell University) · doi:10.3929/ethz-b-000527311 · Cited by 3
An Optimized Dataflow for Mitigating Attention Performance BottlenecksPreprint
Kao, S.-C., Subramanian, S., Agrawal, G., & Krishna, T. (2021) · arXiv (Cornell University) · Cited by 2
Sequential Attention Module for Natural Language ProcessingPreprint
Zhou, M., Ma, J., Yang, H., Jiang, L., & Mo, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03009 · Cited by 4
Exploring a Bilingual Next Word Predictor for a Federated Learning Mobile ApplicationPreprint
Burgos, N. A., Kiš, K., Bakaráč, P., Kvasnica, M., & Licitra, G. (2022) · doi:10.36227/techrxiv.18058682.v2 · Cited by 2
CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors GuidancePreprint
Thwal, C. M., Tun, Y. L., Nguyen, M. N. H., Huh, E.-N., & Hong, C. S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5537956 · Cited by 1
What’s Hidden in a One-layer Randomly Weighted Transformer?
Shen, S., Yao, Z., Kiela, D., Keutzer, K., & Mahoney, M. W. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.231 · Cited by 2
Is Generative AI Ready to Join the Conversation That We Are?
Hornby, R. (2024) · Technophany A Journal for Philosophy and Technology · doi:10.54195/technophany.18134 · Cited by 2
Xducation of Things (XoT): Harnessing AI and Edge Computing to Educate All Things
Nurtantyana, R., Hwang, W., & Hariyanti, U. (2024) · IEEE Access · doi:10.1109/access.2024.3474015 · Cited by 3
Automatic Gender Identification from Text
Litvak, M., Rabaev, I., & Iounkin, V. (2024) · Applied Sciences · doi:10.3390/app142412041 · Cited by 2
P-Distill: Efficient and Effective Prompt Tuning Using Knowledge Distillation
Won, H.-S., Choi, J.-Y., Zaman, N., Aliyeva, D., & Kim, K.-M. (2025) · Applied Sciences · doi:10.3390/app15052420 · Cited by 2
Alshamrani, S. (2025) · IEEE Transactions on Sustainable Computing · doi:10.1109/tsusc.2025.3605796 · Cited by 1
Elastic Architecture Search for Efficient Language Models
Wang, S. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11210235 · Cited by 1
SAFE Natural Language Processing
Babaei, G., Giudice, O., Giudici, P., & Maggi, A. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228343 · Cited by 1
Invited Paper: Optimizing Privacy-Preserving Primitives to Support LLM-Scale Applications
Jandali, Y., Zhang, R., Sheybani, N., & Koushanfar, F. (2025) · doi:10.1109/iccad66269.2025.11240746 · Cited by 1
DPKT: A Novel Knowledge Tracing Framework Based on Distilling Pre-Trained Language Models
Xu, G., & Liang, K. (2025) · IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA) · doi:10.1109/ispa67752.2025.00094 · Cited by 1
A Cybersecurity NER Method Based on Hard and Easy Labeled Training Data Discrimination
Ye, L., Wu, Y., Zhang, H., & Ge, M. (2025) · Sensors · doi:10.3390/s25247627 · Cited by 1
Lau, W., & Isa, S. M. (2025) · Proceeding of the Electrical Engineering Computer Science and Informatics · doi:10.1109/eecsi67060.2025.11290133 · Cited by 1
Liu, X., Yang, P., Liu, X., Wang, Z., & Bai, Z. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3633250 · Cited by 1
Mirkute, D., Arumugam, M., Jaulkar, A., Yadav, P., Ingale, K., & Neelima, N. (2025) · International Conference on Emerging Trends in Networks and Computer Communications (ETNCC) · doi:10.1109/etncc66224.2025.11299777 · Cited by 1
A Multimodal Privacy Filtering System Using Deep Learning for Visual-Audio Input Streams
Kim, K., Park, S.-W., Lee, H.-J., Choi, H., & Buu, S.-J. (2026) · IEEE Access · doi:10.1109/access.2026.3670925 · Cited by 1
A Privacy-Preserving On-Device Framework for Telecom Fraud Detection*
Zhou, X., Zhao, S., Sun, J., & Wang, H. (2025) · IEEE Annual Congress on Artificial Intelligence of Things (AIoT) · doi:10.1109/aiot66900.2025.00147 · Cited by 1
Lim, Z. Y., Pang, Y. H., Jun, E. C. K., Yin, O. S., & Ling, G. F. (2026) · Future Internet · doi:10.3390/fi18030157 · Cited by 1
The Role of Small Language Models in Edge Computing and On-Device AI: A Review
J, J. S. D., Lankesh, M., & Vargheese, A. G. (2026) · International Conference on Intelligent Systems for Communication, IoT and Security (ICISCoIS) · doi:10.1109/iciscois62701.2026.11447893 · Cited by 1
Cross-layer Attention Sharing for Pre-trained Large Language Models
Mu, Y., Wu, Y., Fan, Y., Wang, C. Q., Li, H., Zeng, J., He, Q., Yang, M., Meng, F., Zhou, J., et al. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.616 · Cited by 25
Hasan, M. S., Ahmed, A., Jabe, M. M. N., & Pirzado, F. A. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00560-6 · Cited by 1
Ferro, E., Ottaviano, A., Lammie, C., İslamoğlu, G., Garofalo, A., Conti, F., Boesch, T., Benini, L., Sebastian, A., & Boybat, I. (2026) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2026.3692104 · Cited by 1
Zeng, C., Jiang, Y., Gao, G., Wang, S., Danier, D., Zhu, B., Rudinac, S., Bull, D., & Zhang, F. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810586 · Cited by 1
Ferro, E., Benmeziane, H., & Boybat, I. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3706975 · Cited by 1
Eskandarian, M., Rabbani, M., Kaniyamattam, A., Nejati, F., Mirani, M., Piya, G., Opushnyev, I., Ghorbani, A. A., & Dadkhah, S. (2026) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2026.104552 · Cited by 1
LIME: Lightweight Multi-head Early-exit Transformer for Network Intrusion Detection
Sofi, I. B., Agarwal, A., & Kaur, K. (2026) · International Conference on Communications · doi:10.1109/icc59461.2026.11587340 · Cited by 1
Pháp, H. C. (2026) · Journal of Information and Telecommunication · doi:10.1080/24751839.2026.2713250 · Cited by 1
Zhang, Y., Yu, J., Wang, K., Yin, Y., Chen, C., & Liu, Q. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60457-8_43 · Cited by 1
Cost-effective Deployment of BERT Models in Serverless Environment
Šuppa, M., Benešová, K., & Švec, A. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-industry.24 · Cited by 4
Chinese Electronic Medical Record Named Entity Recognition based on FastBERT method
Tuo, J., Liu, Z., Chen, Q., Ma, X., & Wang, Y. (2021) · Chinese Control and Decision Conference (CCDC) · doi:10.1109/ccdc52312.2021.9602374 · Cited by 2
Chemical transformer compression for accelerating both training and inference of molecular modeling
Yu, Y., & Börjesson, K. (2022) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ac99ba · Cited by 2
Multi-stage Distillation Framework for Cross-Lingual Semantic Similarity Matching
Ding, K., Liu, W., Fang, Y., Zhao, Z., Ju, Q., Yang, X., Tian, R., Zhu, T., Liu, H., Guo, H. W., et al. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.167 · Cited by 1
RA-KD: Random Attention Map Projection for Knowledge Distillation
Zhang, L., Chen, Y., Cao, Y., & Zhao, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-4752-2_48 · Cited by 1
PILE: Pairwise Iterative Logits Ensemble for Multi-Teacher Labeled Distillation
Cai, L., Zhang, L., Ma, D., Fan, J., Shi, D., Wu, Y., Cheng, Z., Gu, S., & Yin, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-industry.60 · Cited by 1
Efficient Two-Stage Progressive Quantization of BERT
Le, C., Ardakani, A., Ardakani, A., Zhang, H., Chen, Y., Clark, J. J., Meyer, B., & Gross, W. J. (2022) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2022.sustainlp-1.2 · Cited by 1
Mozafari, S. H., Clark, J. J., Gross, W. J., & Meyer, B. H. (2023) · IEEE International Conference on Application-specific Systems, Architectures and Processors (ASAP) · doi:10.1109/asap57973.2023.00024 · Cited by 1
Kong, J., Wang, J., & Zhang, X. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3331080 · Cited by 1
Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules
Xiao, C., Luo, Y., Zhang, W., Zhang, P., Han, X., Lin, Y., Zhang, Z., Xie, R., Liu, Z., Sun, M., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.666 · Cited by 1
Leap-of-Thought: Accelerating Transformers via Dynamic Token Routing
Kim, Y., Kim, J., Park, J.-H., Lee, M., & Lee, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.976 · Cited by 1
Qi, W.-K., Guo, X., & Du, H. (2023) · International Conference on Big Data Computing and Communications (BigCom) · doi:10.1109/bigcom61073.2023.00044 · Cited by 1
Location Aware Modular Biencoder for Tourism Question Answering
Li, H., Tomko, M., & Baldwin, T. (2023) · doi:10.18653/v1/2023.findings-ijcnlp.9 · Cited by 1
Sensi-Bert: Towards Sensitivity Driven Fine-Tuning for Parameter-Efficient Language Model
Kundu, S., Sridhar, S. N., Szankin, M., & Sundaresan, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447812 · Cited by 2
Scaling Up the Transformers: A Survey of Training and Inference Optimization Techniques
Jahangir, H., Goel, S. K., & Khurana, S. (2024) · International Conference on Electrical Electronics and Computing Technologies (ICEECT) · doi:10.1109/iceect61758.2024.10739061 · Cited by 1
Compressing Neural Networks using Learnable 1D Non-Linear Functions
Singh, G., & Bazargan, K. (2024) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3705926 · Cited by 1
A transformer-based model for next disease prediction using electronic health records
Makarov, N., & Lipkovich, M. (2025) · The European Physical Journal Special Topics · doi:10.1140/epjs/s11734-024-01447-1 · Cited by 2
A low functional redundancy-based network slimming method for accelerating deep neural networks
Fang, Z. N., & Yin, B. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2024.12.118 · Cited by 1
ETBench: Characterizing Hybrid Vision Transformer Workloads Across Edge Devices
Zhou, Y., Tian, Z., Yang, W., Zhang, T., Ye, J., Han, C., Liu, T., & Zhang, F. (2025) · IEEE Transactions on Computers · doi:10.1109/tc.2025.3543697 · Cited by 1
Dataset Distillation with Attention Labels for Fine-tuning BERT
Maekawa, A., Kobayashi, N., Funakoshi, K., & Okumura, M. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.283 · Cited by 1
Zhang, J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., Xie, Q., Xie, G., & Gong, X. (2025) · The VLDB Journal · doi:10.1007/s00778-025-00919-7 · Cited by 1
Deep Learning Architectures for Text Classification
Desai, C. G. (2025) · doi:10.38124/ijisrt/25may1682 · Cited by 2
The Chinese Remainder Theorem for Compact, Task-Precise, Efficient and Secure Word Embeddings
Thaine, P., & Penn, G. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.306 · Cited by 1
DAP-BERT: Differentiable Architecture Pruning of BERT
Yau, C.-Y., Bai, H., King, I., & Lyu, M. R.-T. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-92185-9_30 · Cited by 1
LightBERT: A Distilled Chinese BERT Model
Zhang, Y., Li, Y., Xu, P., Xu, R., Li, J., Shi, G., & Hu, F. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-96033-9_5 · Cited by 1
All Mistakes are not Equal: Comprehensive Hierarchy Aware Multilabel Predictions (CHAMP)
Vaswani, A., Samaga, Y., Aggarwal, G., Netrapalli, P., & Hegde, N. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78107-0_17 · Cited by 1
On The Connection of Benford's Law and Neural Networks.Preprint
Sahu, S. K., Java, A., & Shaikh, A. (2021) · arXiv (Cornell University) · Cited by 1
Cost-effective Deployment of BERT Models in Serverless EnvironmentPreprint
Benešová, K., Švec, A., & Šuppa, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.10673 · Cited by 1
MergeDistill: Merging Pre-trained Language Models using DistillationPreprint
Khanuja, S., Johnson, M., & Talukdar, P. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02834 · Cited by 1
Liu, P., Gao, Z.-F., Zhao, W. X., Xie, Z. Y., Lu, Z.-Y., & Wen, J.-R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02205 · Cited by 1
Scheduling Optimization Techniques for Neural Network TrainingPreprint
Oh, H., Oh, H., Kim, H., & Seo, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.00929 · Cited by 1
Exploring a Bilingual Next Word Predictor for a Federated Learning Mobile ApplicationPreprint
Burgos, N. A., Kiš, K., Bakaráč, P., Kvasnica, M., & Licitra, G. (2022) · doi:10.36227/techrxiv.18058682.v1 · Cited by 1
Exploring a Bilingual Next Word Predictor for a Federated Learning Mobile ApplicationPreprint
Burgos, N. A., Kiš, K., Bakaráč, P., Kvasnica, M., & Licitra, G. (2022) · doi:10.36227/techrxiv.18058682 · Cited by 1
A Cohesive Distillation Architecture for Neural Language ModelsPreprint
Wahle, J. P. (2023) · doi:10.22541/au.167528147.79728645/v1 · Cited by 1
A Comprehensive Survey on the Integration of Reinforcement Learning and NLP for Stock Market TradingPreprint
Ferrell, B., & McInnes, B. T. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5135573 · Cited by 1
Qiang, Y., Kumar, S. T. S., Brocanelli, M., & Zhu, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.05624
Annealing Knowledge DistillationPreprint
Jafari, A., Rezagholizadeh, M., Sharma, P., & Ghodsi, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07163 · Cited by 19
Lightweight Cross-Lingual Sentence Representation LearningPreprint
Mao, Z., Gupta, P., Wang, P., Chu, C., Jaggi, M., & Kurohashi, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13856
Why Can You Lay Off Heads? Investigating How BERT Heads TransferPreprint
Chiang, T.-R., & Chen, Y.-N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.07137
Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge DistillationPreprint
Liu, Y., Meng, F., Lin, Z., Wang, W., & Zhou, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05691
Reservoir TransformersPreprint
Shen, S., Baevski, A., Morcos, A. S., Keutzer, K., Auli, M., & Kiela, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15045 · Cited by 17
ICIP at TREC-2020 Deep Learning Track.
Chen, X., He, B., Sun, L., & Sun, Y. (2020) · Text REtrieval Conference
LV-BERT: Exploiting Layer Variety for BERT
Yu, W., Jiang, Z., Chen, F., Hou, Q., & Feng, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.2 · Cited by 1
Design Space Exploration of Sparse Accelerators for Deep Neural Networks.Preprint
Shin, J. H., Shafiee, A. R., Pedram, A., Abdel-Aziz, H. A., Li, L., & Hassoun, J. (2021) · arXiv (Cornell University)
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language ModelsPreprint
Yin, Y., Chen, C., Shang, L., Jiang, X., Chen, X., & Liu, Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.13686 · Cited by 10
Weight Squeezing: Reparameterization for Knowledge Transfer and Model CompressionPreprint
Chumachenko, A., Gavrilov, D., Balagansky, N., & Kalaidin, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06993 · Cited by 1
New Students on Sesame Street: What Order-Aware Matrix Embeddings Can Learn from BERT.Preprint
Galke, L. P. A., Cuber, I., Meyer, C. F. J., Nölscher, H. F., Sonderecker, A., & Scherp, A. (2021) · arXiv (Cornell University)
When in Doubt, Summon the Titans: Efficient Inference with Large ModelsPreprint
Rawat, A. S., Zaheer, M., Menon, A. K., Ahmed, A. A., & Kumar, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10305 · Cited by 6
Sparse Distillation: Speeding Up Text Classification by Using Bigger ModelsPreprint
Ye, Q., Khabsa, M., Lewis, M., Wang, S., Ren, X., & Jaech, A. (2021) · arXiv (Cornell University)
Knowledge Distillation with Noisy Labels for Natural Language UnderstandingPreprint
Bhardwaj, S., Ghaddar, A., Rashid, A., Bibi, K., Li, C., Ghodsi, A., Langlais, P., & Rezagholizadeh, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10147 · Cited by 1
Edge-Cloud Polarization and Collaboration: A Comprehensive SurveyPreprint
Yao, J., Zhang, S., Yang, Y., Wang, F., Ma, J., Zhang, J., Chu, Y., Ji, L., Jia, K., Shen, T., et al. (2021) · arXiv (Cornell University)
Self-citationBrian Roark, A., & Gutkin, A. (2022) · Workshop on Speech and Language Processing for Assistive Technologies · doi:10.18653/v1/2022.slpat-1.1 · Cited by 1
Architectures for Machine Learning
Yang, Y., Chen, C., & Wang, Z. (2022) · Springer eBooks · doi:10.1007/978-981-15-6401-7_12-1
Self-citationREDIT: Resilient Distributed Text-to-Speech at Edge Networks
Li, T., Seferoğlu, H., & Koyuncu, E. (2022) · Globecom · doi:10.1109/globecom48099.2022.10001480
A Corporate Public Opinion Classification Method Based on Multi-teacher Knowledge Distillation
Chen, X., Wenjue, X., & Li, W. (2023) · IEEE International Conference on Power, Electronics and Computer Applications (ICPECA) · doi:10.1109/icpeca56706.2023.10076112
Representation Learning for Texts and Graphs
Galke, L. P. A. (2023) · doi:10.21941/kcss/2023/1
The EsnTorch Library: Efficient Implementation of Transformer-Based Echo State Networks
Cabessa, J., Hernault, H., Lamonato, Y., Rochat, M., & Levy, Y. Z. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-1648-1_20
SAMNet: Semantic Aware Multimodal Network for Emoji Drawing Classification
Gothe, S. V., Khurana, R., Vachhani, J. R., Rakshith, S., & Kashyap, P. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-31417-9_10
Falaki, A. A., & Gras, R. (2023) · Machine Learning and Knowledge Extraction · doi:10.3390/make5030045
Operator Selection and Ordering in a Pipeline Approach to Efficiency Optimizations for Transformers
Xin, J., Tang, R., Jiang, Z., Yu, Y., & Lin, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.180
Metadata
- Authors
- Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., & Zhou, D.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 2158-2170
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3034457371
- Cited by
- 757
- References
- 54
- Citations by year
- 2026: 94 · 2025: 134 · 2024: 98 · 2023: 128 · 2022: 119 · 2021: 140 · 2020: 43
- FWCI
- 41.07
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- MobileBERT, BERT, model compression, knowledge distillation, teacher-student learning, self-attention, feedforward neural networks, mobile devices, resource-constrained deployment, natural language inference, question answering, adhesives, SQuAD
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
