The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
Open accessAbstract
Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds of millions of parameters.However, these models suffer from heavy model sizes and high latency such that they cannot be deployed to resourcelimited mobile devices.In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model.Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to various downstream NLP tasks via simple fine-tuning.Basically, MobileBERT is a thin version of BERT LARGE , while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks.To train MobileBERT, we first train a specially designed teacher model, an invertedbottleneck incorporated BERT LARGE model.Then, we conduct knowledge transfer from this teacher to MobileBERT.Empirical studies show that MobileBERT is 4.3× smaller and 5.5× faster than BERT BASE while achieving competitive results on well-known benchmarks.On the natural language inference tasks of GLUE, MobileBERT achieves a GLUE score of 77.7 (0.6 lower than BERT BASE ), and 62 ms latency on a Pixel 4 phone.On the SQuAD v1.1/v2.0question answering task, MobileBERT achieves a dev F1 score of 90.0/79.2(1.5/2.1 higher than BERT BASE ).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“To train MobileBERT, first the authors trained a specially designed teacher model, and then, they transferred the knowledge from the teacher to MobileBERT, which is 4.3× smaller and 5.5× faster than the general BERT model.”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [57]
The cited source says what the claim attributes to it.
In the citing paper
“The distilled MobileBERT model is 4.3 × smaller than the BERT-base network and 5.5 × faster on a Pixel 4 mobile phone.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“MobileBERT [67] is a compressed version of the BERT-base model, and it runs 5.5 ×”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [67]
In the citing paper
“Moreover, models such as MobileBERT and SqueezeBERT employ bottleneck layers and grouped convolutions, leading to a 3× to 4× reduction in latency while maintaining >90% of the original model’s accuracy.”
From Tiny Language Models for Automation and Control: Overview, Potential Applications, and Future Research Directions (2025) · cites this paper as [68]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
(2010) · International Conference on Machine Learning · Cited by 13,995
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Research works printed but not linked: 2
- William B Dolan and Chris. Brockett. 2005. Auto- matically constructing a corpus of sentential para- phrases. In Proceedings of the International Work- shop on Paraphrasing. 7 https://tensorflow.org/lite/models/ bert_qa/overview
- Ganesh Jawahar, Benoît Sagot, Djamé Seddah, Samuel Unicomb, Gerardo Iñiguez, Márton Karsai, Yannick Léo, Márton Karsai, Carlos Sarraute, Éric Fleury, et al. 2019. What does bert learn about the structure of language? In 57th Annual Meeting of the Associa- tion for Computational Linguistics (ACL), Florence, Italy.
Cited by
Sorted by citations
Items 201–300 of 758
Improving question answering performance using knowledge distillation and active learning
Boreshban, Y., Mirbostani, S. M., Ghassem-Sani, G., Mirroshandel, S. A., & Amiriparian, S. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106137 · Cited by 22
Character-level HyperNetworks for Hate Speech Detection
Wullach, T., Adler, A., & Minkov, E. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117571 · Cited by 18
Chinese Named Entity Recognition Based on Knowledge Based Question Answering System
Yin, D., Cheng, S., Pan, B., Qiao, Y., Zhao, W., & Wang, D. (2022) · Applied Sciences · doi:10.3390/app12115373 · Cited by 15
X-Education: Education of All Things with AI and Edge Computing—One Case Study for EFL Learning
Hwang, W., & Nurtantyana, R. (2022) · Sustainability · doi:10.3390/su141912533 · Cited by 15
Textual Pre-Trained Models for Gender Identification Across Community Question-Answering Members
Schwarzenberg, P., & Figueroa, A. (2023) · IEEE Access · doi:10.1109/access.2023.3235735 · Cited by 17
Griffin: Rethinking Sparse Optimization for Deep Learning Architectures
Shin, J. H., Shafiee, A. R., Pedram, A., Abdel-Aziz, H. A., Li, L., & Hassoun, J. (2022) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca53966.2022.00068 · Cited by 17
Oh, H., Lee, J., Kim, H., & Seo, J. (2022) · European Conference on Computer Systems · doi:10.1145/3492321.3519563 · Cited by 19
PipeBERT: High-throughput BERT Inference for ARM Big.LITTLE Multi-core Processors
Chang, H.-Y., Mozafari, S. H., Chen, C., Clark, J. J., Meyer, B. H., & Gross, W. J. (2023) · Journal of Signal Processing Systems · doi:10.1007/s11265-022-01814-y · Cited by 14
SECDA-TFLite: A toolkit for efficient development of FPGA-based DNN accelerators for edge inference
Haris, J., Gibson, P., Cano, J., Agostini, N. B., & Kaeli, D. R. (2022) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2022.11.005 · Cited by 14
SKDBERT: Compressing BERT via Stochastic Knowledge Distillation
Ding, Z., Guoqing, J., Zhang, S., Guo, L., & Lin, W. T. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25902 · Cited by 14
Distilled BERT Model in Natural Language Processing
Vakili, Y. Z., Fallah, A., & Sajedi, H. (2024) · International Conference on Computer and Knowledge Engineering · doi:10.1109/iccke65377.2024.10874673 · Cited by 16
Generative AI at the Edge: Challenges and Opportunities
Reddi, V. J. (2025) · Queue · doi:10.1145/3733702 · Cited by 17
Attention Temperature Matters in Abstractive Summarization Distillation
Zhang, S., Zhang, X., Bao, H., & Wei, F. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.11 · Cited by 15
Multi-Grained Attention Representation With ALBERT for Aspect-Level Sentiment Classification
Chen, Y., Kong, L., Wang, Y., & Kong, D. (2021) · IEEE Access · doi:10.1109/access.2021.3100299 · Cited by 14
Improving task-agnostic BERT distillation with layer mapping search
Jiao, X., Chang, H., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.07.050 · Cited by 14
Question Answering with BERT: designing a 3D virtual avatar for Cultural Heritage exploration
Farella, M., Chiazzese, G., & Bosco, G. L. (2022) · IEEE Mediterranean Electrotechnical Conference (MELECON) · doi:10.1109/melecon53508.2022.9843028 · Cited by 15
Optimizing BFloat16 Deployment of Tiny Transformers on Ultra-Low Power Extreme Edge SoCs
Dequino, A., Bompani, L., Benini, L., & Conti, F. (2025) · Journal of Low Power Electronics and Applications · doi:10.3390/jlpea15010008 · Cited by 15
CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer
Wang, Y., Wang, F., Dong, J., & Luo, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28376 · Cited by 19
Multimodal fine-grained grocery product recognition using image and OCR text
Pettersson, T., Riveiro, M., & Löfström, T. (2024) · Machine Vision and Applications · doi:10.1007/s00138-024-01549-9 · Cited by 22
Anatomizing Deep Learning Inference in Web Browsers
Wang, Q., Jiang, S., Chen, Z., Cao, X., Li, Y., Li, A., Ma, Y., Cao, T., & Liu, X. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3688843 · Cited by 29
SinKD: Sinkhorn Distance Minimization for Knowledge Distillation
Cui, X., Qin, Y., Gao, Y., Zhang, E., Xu, Z., Wu, T., Li, K., Sun, X., Zhou, W., & Li, H. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3501335 · Cited by 15
BERTino: an Italian DistilBERT model
Muffo, M., & Bertino, E. (2020) · Accademia University Press eBooks · doi:10.4000/books.aaccademia.8748 · Cited by 18
Searching for Efficient Neural Architectures for On-Device ML on Edge TPUs
Akin, B., Gupta, S., Long, Y., Spiridonov, A., Wang, Z., White, M., Xu, H., Zhou, P., & Zhou, Y. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00300 · Cited by 16
Adversarial Data Augmentation for Task-Specific Knowledge Distillation of Pre-trained Transformers
Zhang, M., Naresh, N. U., & He, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21423 · Cited by 14
Gibson, P., & Cano, J. (2022) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1145/3559009.3569682 · Cited by 13
FedCD: Personalized Federated Learning via Collaborative Distillation
Ahmad, S., & Aral, A. (2022) · IEEE/ACM International Conference on Utility and Cloud Computing (UCC) · doi:10.1109/ucc56403.2022.00036 · Cited by 15
FlexiBERT: Are Current Transformer Architectures too Homogeneous and Rigid?
Tuli, S., Dedhia, B., Tuli, S., & Jha, N. K. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13942 · Cited by 16
On the Effects of Transformer Size on In- and Out-of-Domain Calibration
Dan, S., & Roth, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.180 · Cited by 14
Joint Inference Offloading and Model Caching for Small and Large Language Model Collaboration
Xu, X., Feng, G., Liu, Y.-J., Qin, S., Wang, J., & Wang, Y. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3608303 · Cited by 13
LIDSNet: A Lightweight on-device Intent Detection model using Deep Siamese Network
Agarwal, V., Shivnikar, S. D., Ghosh, S. K., Arora, H., & Saini, Y. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00182 · Cited by 13
AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression
Wu, S., Chen, H., Quan, X., Wang, Q., & Wang, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.471 · Cited by 30
ReAugKD: Retrieval-Augmented Knowledge Distillation For Pre-trained Language Models
Zhang, J., Muhamed, A., Anantharaman, A., Wang, G., Chen, C., Zhong, K., Cui, Q., Xu, Y., Zeng, B., Chilimbi, T. M., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.97 · Cited by 17
Trace-Based Performance Analysis for Deep Learning in Edge Container Environments
Park, S., & Bahn, H. (2023) · International Conference on Fog and Mobile Edge Computing (FMEC) · doi:10.1109/fmec59375.2023.10306027 · Cited by 12
Distilling Wisdom: A Review on Optimizing Learning From Massive Language Models
Zhang, D., Listiyani, D., Singh, P., & Mohanty, M. (2025) · IEEE Access · doi:10.1109/access.2025.3554586 · Cited by 18
ARBML: Democratizing Arabic Natural Language Processing Tools
Alyafeai, Z., & Al-shaibani, M. S. (2020) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/2020.nlposs-1.2 · Cited by 12
Tahaei, M. S., Charlaix, E., Nia, V. P., Ghodsi, A., & Rezagholizadeh, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.154 · Cited by 18
Moosavi, N. S., Delfosse, Q., Kersting, K., & Gurevych, I. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.274 · Cited by 12
Exploring the Performance and Efficiency of Transformer Models for NLP on Mobile Devices
Panopoulos, I., Nikolaidis, S., Venieris, S. I., & Venieris, I. S. (2023) · IEEE Symposium on Computers and Communications (ISCC) · doi:10.1109/iscc58397.2023.10217850 · Cited by 16
Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding
Du, M., Mukherjee, S., Cheng, Y., Shokouhi, M., Xia, H., & Awadallah, A. H. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.129 · Cited by 30
Towards Efficient NLP: A Standard Evaluation and A Strong Baseline
Liu, X., Sun, T., He, J., Wu, J., Wu, L., Zhang, X., Jiang, H., Cao, Z., Huang, X., & Qiu, X. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.240 · Cited by 18
Wang, J., Wu, S., Qu, Z., Shen, J., Memon, S. A., Skitmore, M., & Ma, L. (2025) · Engineering Construction & Architectural Management · doi:10.1108/ecam-08-2025-1273 · Cited by 14
TaughtNet: Learning Multi-Task Biomedical Named Entity Recognition From Single-Task Teachers
Moscato, V., Postiglione, M., Sansone, C., & Sperlí, G. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3244044 · Cited by 12
Kim, M., Lee, S.-H., Hong, S.-J., Chang, D.-S., & Choi, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.450 · Cited by 13
Explainable Attention Pruning: A Metalearning-Based Approach
Rajapaksha, P., & Crespi, N. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3363686 · Cited by 12
Efficiency at scale: Investigating the performance of diminutive language models in clinical tasks
Taylor, N., Ghose, U., Rohanian, O., Nouriborji, M., Kormilitzin, A., Clifton, D. A., & Nevado‐Holgado, A. J. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.103002 · Cited by 14
Automated Text Scoring in the Age of Generative AI for the GPU-poor
Ormerod, C. M., & Kwako, A. (2024) · Chinese/English Journal of Educational Measurement and Evaluation · doi:10.59863/okuu1904 · Cited by 20
Evaluating BERT-based language models for detecting misinformation
Anggrainingsih, R., Hassan, G. M., & Datta, A. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11101-z · Cited by 14
Peralta, A., Olivas, J. Á., Romero, F. P., & Navarro-Illana, P. (2025) · Mathematics · doi:10.3390/math13071055 · Cited by 11
A Survey on the Impact of Pre-Trained Language Models in Sentiment Classification Task
Gautam, H., Gaur, A., & Yadav, D. K. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00805-z · Cited by 14
End-To-End Neural Coreference Resolution Revisited: A Simple Yet Effective Baseline
Lai, T., Bui, T., & Kim, D. S. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746254 · Cited by 14
Compression Methods for Transformers in Multidomain Sentiment Analysis
Korczyński, W., & Kocoń, J. (2022) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw58026.2022.00062 · Cited by 11
Liu, P., Gao, Z.-F., Zhao, W. X., Xie, Z. Y., Lu, Z.-Y., & Wen, J.-R. (2021) · doi:10.18653/v1/2021.acl-long.418 · Cited by 27
Tahaei, M. S., Charlaix, E., Nia, V. P., Ghodsi, A., & Rezagholizadeh, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.06243 · Cited by 27
Maheen, S. M., Faisal, M. R., & Karim, M. R. R. A. M. S. (2022) · Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo) · doi:10.18653/v1/2022.deeplo-1.20 · Cited by 12
An in-depth analysis of pre-trained embeddings for entity resolution
Zeakis, A., Papadakis, G., Skoutas, D., & Koubarakis, M. (2024) · The VLDB Journal · doi:10.1007/s00778-024-00879-4 · Cited by 10
Gao, J. G., Lin, S., Wang, S., Kou, Y., Li, Z., Li, L., Zhang, C., Zhang, X., Wang, Y., & Hu, W. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-024-02327-w · Cited by 12
Climate-Resilient Agriculture: Leveraging Language Models for Mitigation and Adaptation
Rao, S., & Ranganath, P. (2025) · Environmental earth sciences · doi:10.1007/978-3-031-75968-0_19 · Cited by 10
A Comparative Analysis of Compression and Transfer Learning Techniques in DeepFake Detection Models
Karathanasis, A., Violos, J., & Kompatsiaris, I. Y. (2025) · Mathematics · doi:10.3390/math13050887 · Cited by 16
Mayya, V., Kamath, S. S., & Sugumaran, V. (2021) · IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB) · doi:10.1109/cibcb49929.2021.9562815 · Cited by 13
Lifting the Curse of Capacity Gap in Distilling Language Models
Zhang, C., Yang, Y., Liu, J., Wang, J., Xian, Y., Wang, B., & Song, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.249 · Cited by 23
Self-citationCybersmish: A Proactive Approach for Smishing Detection and Prevention using Machine Learning
Kumarasiri, W. L. T. T. N., Siriwardhana, M. K. J. C., Suraweera, S. A. D. S. L., Senarathne, A. N., & Harshanath, S. M. B. (2023) · International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC) · doi:10.1109/i-smac58438.2023.10290228 · Cited by 10
Self-Guidance: Improve Deep Neural Network Generalization via Knowledge Distillation
Zheng, Z., & Peng, X. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00351 · Cited by 10
VIRT: Improving Representation-based Text Matching via Virtual Interaction
Li, D., Yang, Y., Tang, H., Liu, J., Wang, Q., Wang, J., Xu, T., Wu, W., & Chen, E. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.59 · Cited by 10
Self-citationLEALLA: Learning Lightweight Language-agnostic Sentence Embeddings with Knowledge Distillation
Mao, Z., & Nakagawa, T. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.138 · Cited by 13
Real-Time Execution of Large-scale Language Models on MobilePreprint
Niu, W., Kong, Z., Yuan, G., Jiang, W., Guan, J., Ding, C., Zhao, P., Liu, S., Ren, B., & Wang, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.06823 · Cited by 14
Knowledge Distillation Across Vision and Language
Fang, Z., & Yang, Y. (2023) · Studies in computational intelligence · doi:10.1007/978-3-031-32095-8_3 · Cited by 9
Self-citationEdgeTran: Device-Aware Co-Search of Transformers for Efficient Inference on Mobile Edge Platforms
Tuli, S., & Jha, N. K. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3328287 · Cited by 16
Efficient Transformer Knowledge Distillation: A Performance Review
Brown, N., Williamson, A., Anderson, T., & Lawrence, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.6 · Cited by 12
Vision Transformer-based overlay processor for Edge Computing
Liu, F., Fan, Z., Hu, W., Xu, D. G., Peng, M., He, S., & He, Y. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111421 · Cited by 10
Towards smarter hiring: resume parsing and ranking with YOLOv5 and DistilBERT
Kinger, S., Kinger, D., Thakkar, S., & Bhake, D. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18778-9 · Cited by 13
Reparameterization of Lightweight Transformer for On-Device Speech Emotion Recognition
Zhang, Z., Dong, Z., Xu, W., & Han, J. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3483232 · Cited by 10
Wiese, P., İslamoğlu, G., Scherer, M., Macan, L., Jung, V. J.-B., Burrello, A., Conti, F., & Benini, L. (2025) · IEEE Design & Test · doi:10.1109/mdat.2025.3527371 · Cited by 9
Liu, S., Radway, R. M., Wang, X., Moro, F., Nodin, J.-F., Jana, K., Yan, L., Du, S., Upton, L. R., Chen, W.-C., et al. (2025) · IEEE Transactions on Electron Devices · doi:10.1109/ted.2025.3556113 · Cited by 10
A Flexible Template for Edge Generative AI With High-Accuracy Accelerated Softmax and GELU
Belano, A., Tortorella, Y., Garofalo, A., Benini, L., Rossi, D., & Conti, F. (2025) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2025.3562734 · Cited by 13
Bad-MFL: A Cross-Modality Bi-Trigger Backdoor Attack Against Multimodal Federated Learning
Lai, Y., Wu, L., Lin, H., & Liu, J. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3589136 · Cited by 9
ResQConnect: An AI-Powered Multi-Agentic Platform for Human-Centered and Resilient Disaster Response
Aththanayake, S., Mallikarachchi, C., Wickramasinghe, J., Kugarajah, S., Meedeniya, D., & Pradhan, B. (2026) · Sustainability · doi:10.3390/su18021014 · Cited by 9
Multidomain Pretrained Language Models for Green NLP
Maronikolakis, A., & Schütze, H. (2021) · Cited by 9
An Empirical Study on Adaptive Inference for Pretrained Language Model
Liu, W., Zhao, X. W., Zhao, Z., Ju, Q., Yang, X., & Lü, W. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3114188 · Cited by 9
Xie, R., Zhang, S., Wang, R., Xia, F., & Lin, L. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3481917 · Cited by 9
Improving Pre-trained Language Models
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_3 · Cited by 9
Learning to Imagine: Distillation-Based Interactive Context Exploitation for Dialogue State Tracking
Guo, J., Shuang, K., Zhang, K., Liu, Y., Li, J., & Wang, Z. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26510 · Cited by 9
Fast Vocabulary Transfer for Language Model Compression
Gee, L., Zugarini, A., Rigutini, L., & Torroni, P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-industry.41 · Cited by 22
Task-specific Compression for Multi-task Language Models using Attribution-based Pruning
Yang, N., Jang, Y., Lee, H., Jeong, S., & Jung, K. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.43 · Cited by 16
Model Compression for Domain Adaptation through Causal Effect Estimation
Rotman, G., Feder, A., & Reichart, R. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00431 · Cited by 9
Larson, S., Singh, N., Maheshwari, S., Stewart, S., & Krishnaswamy, U. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86159-9_30 · Cited by 10
Cross-domain knowledge distillation for text classification
Zhang, S., Jiang, L., & Tan, J. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.08.061 · Cited by 9
Hazarika, D., Chanda, S., & Gupta, C. N. (2022) · IEEE-EMBS Conference on Biomedical Engineering and Sciences · doi:10.1109/iecbes54088.2022.10079412 · Cited by 9
TransCODE: Co-Design of Transformers and Accelerators for Efficient Training and Inference
Tuli, S., & Jha, N. K. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3283443 · Cited by 11
Tailoring Instructions to Student’s Learning Levels Boosts Knowledge Distillation
Ren, Y., Zhong, Z., Shi, X., Zhu, Y., Yuan, C., & Li, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.111 · Cited by 15
XtremeDistilTransformers: Task Transfer for Task-agnostic DistillationPreprint
Mukherjee, S., Awadallah, A. H., & Gao, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.04563 · Cited by 26
DKM: Differentiable K-Means Clustering Layer for Neural Network CompressionPreprint
Cho, M., Alizadeh-Vahid, K., Adya, S. N., & Rastegari, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.12659 · Cited by 9
Ahanger, M. M., Wani, M. A., & Palade, V. (2024) · Knowledge · doi:10.3390/knowledge4030022 · Cited by 10
A DRL-Based Real-Time Video Processing Framework in Cloud–Edge Systems
Fu, X., Pan, L., & Liu, S. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3451496 · Cited by 9
Choi, J., Yang, J.-Y., & Chang, J. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3463491 · Cited by 9
Speech and multilingual natural language framework for speaker change detection and diarization
Anidjar, O. H., Estève, Y., Hajaj, C., Dvir, A. Z., & Lapidot, I. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119238 · Cited by 8
Distilling Monolingual Models from Large Multilingual Transformers
Singh, P., Clercq, O. D., & Lefever, E. (2023) · Electronics · doi:10.3390/electronics12041022 · Cited by 9
TCAMixer: A lightweight Mixer based on a novel triple concepts attention mechanism for NLP
Liu, X., Tang, H., Zhao, J., Dou, Q., & Lu, M. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106471 · Cited by 8
Wang, X., Weissweiler, L., Schütze, H., & Plank, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.157 · Cited by 11
GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model
Tan, S., Tam, W. L., Wang, Y., Gong, W., Zhao, S., Zhang, P., & Tang, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.15 · Cited by 17
The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment
Fernandez, J., Kahn, J., Na, C., Bisk, Y., & Strubell, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.98 · Cited by 11
Metadata
- Authors
- Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., & Zhou, D.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 2158-2170
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3034457371
- Cited by
- 757
- References
- 54
- Citations by year
- 2026: 94 · 2025: 134 · 2024: 98 · 2023: 128 · 2022: 119 · 2021: 140 · 2020: 43
- FWCI
- 41.07
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- MobileBERT, BERT, model compression, knowledge distillation, teacher-student learning, self-attention, feedforward neural networks, mobile devices, resource-constrained deployment, natural language inference, question answering, adhesives, SQuAD
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
