The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
Open accessAbstract
Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds of millions of parameters.However, these models suffer from heavy model sizes and high latency such that they cannot be deployed to resourcelimited mobile devices.In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model.Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to various downstream NLP tasks via simple fine-tuning.Basically, MobileBERT is a thin version of BERT LARGE , while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks.To train MobileBERT, we first train a specially designed teacher model, an invertedbottleneck incorporated BERT LARGE model.Then, we conduct knowledge transfer from this teacher to MobileBERT.Empirical studies show that MobileBERT is 4.3× smaller and 5.5× faster than BERT BASE while achieving competitive results on well-known benchmarks.On the natural language inference tasks of GLUE, MobileBERT achieves a GLUE score of 77.7 (0.6 lower than BERT BASE ), and 62 ms latency on a Pixel 4 phone.On the SQuAD v1.1/v2.0question answering task, MobileBERT achieves a dev F1 score of 90.0/79.2(1.5/2.1 higher than BERT BASE ).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“To train MobileBERT, first the authors trained a specially designed teacher model, and then, they transferred the knowledge from the teacher to MobileBERT, which is 4.3× smaller and 5.5× faster than the general BERT model.”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [57]
The cited source says what the claim attributes to it.
In the citing paper
“The distilled MobileBERT model is 4.3 × smaller than the BERT-base network and 5.5 × faster on a Pixel 4 mobile phone.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“MobileBERT [67] is a compressed version of the BERT-base model, and it runs 5.5 ×”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [67]
In the citing paper
“Moreover, models such as MobileBERT and SqueezeBERT employ bottleneck layers and grouped convolutions, leading to a 3× to 4× reduction in latency while maintaining >90% of the original model’s accuracy.”
From Tiny Language Models for Automation and Control: Overview, Potential Applications, and Future Research Directions (2025) · cites this paper as [68]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
(2010) · International Conference on Machine Learning · Cited by 13,995
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Research works printed but not linked: 2
- William B Dolan and Chris. Brockett. 2005. Auto- matically constructing a corpus of sentential para- phrases. In Proceedings of the International Work- shop on Paraphrasing. 7 https://tensorflow.org/lite/models/ bert_qa/overview
- Ganesh Jawahar, Benoît Sagot, Djamé Seddah, Samuel Unicomb, Gerardo Iñiguez, Márton Karsai, Yannick Léo, Márton Karsai, Carlos Sarraute, Éric Fleury, et al. 2019. What does bert learn about the structure of language? In 57th Annual Meeting of the Associa- tion for Computational Linguistics (ACL), Florence, Italy.
Cited by
Sorted by citations
Items 301–400 of 758
A Comprehensive Review of Knowledge Distillation- Methods, Applications, and Future Directions
Zhu, E. Y., Zhao, C., Yang, H., Li, J., Wu, Y., & Ding, R. (2024) · International Journal of Innovative Research in Computer Science & Technology · doi:10.55524/ijircst.2024.12.3.17 · Cited by 8
TLS-SLAM: Gaussian Splatting SLAM Tailored for Large-Scale Scenes
Cheng, S., He, S., Duan, F., & An, N. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3536876 · Cited by 8
Karim, A. A. J., Asad, K. H., & Alam, M. G. R. (2025) · PLoS ONE · doi:10.1371/journal.pone.0315829 · Cited by 10
EdgeBERT: Optimizing On-Chip Inference for Multi-Task NLP.
Tambe, T., Hooper, C. R. C., Pentecost, L., Yang, E.-Y., Donato, M., Sanh, V., Rush, A. M., Brooks, D. M., & Wei, G.-Y. (2020) · Cited by 8
Efficient Relational Sentence Ordering Network
Li, Y., Cui, B., & Zhang, Z. M. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3085738 · Cited by 8
Are Rotten Apples Edible? Challenging Commonsense Inference Ability with Exceptions
Do, N., & Pavlick, E. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.181 · Cited by 8
Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training
Liu, Y., Meng, F., Lin, Z., Fu, P., Cao, Y., Wang, W., & Zhou, J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.428 · Cited by 9
External Knowledge Acquisition for End-to-End Document-Oriented Dialog Systems
Lai, T., Castellucci, G., Kuzi, S., Ji, H., & Rokhlenko, O. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.264 · Cited by 9
Forward Learning of Large Language Models by Consumer Devices
Pau, D. P., & Aymone, F. M. (2024) · Electronics · doi:10.3390/electronics13020402 · Cited by 10
MergeDistill: Merging Language Models using Pre-trained Distillation
Khanuja, S., Johnson, M., & Talukdar, P. P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.254 · Cited by 10
Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation
Chen, C., Yin, Y., Shang, L., Wang, Z., Jiang, X., Chen, X., & Liu, Q. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86365-4_46 · Cited by 9
A Review on Transformer Models: Applications, Taxonomies, Open Issues and Challenges
Passi, N., Raj, M., & Shelke, N. A. (2024) · Asian Conference on Innovation in Technology (ASIANCON) · doi:10.1109/asiancon62057.2024.10838047 · Cited by 8
Muhetaer, M., Meng, X., Zhu, J., Aikebaier, A., Zu, L., & Bai, Y. (2025) · Symmetry · doi:10.3390/sym17111812 · Cited by 7
Boost Transformer-based Language Models with GPU-Friendly Sparsity and Quantization
Yu, C. H., Chen, T., & Gan, Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.15 · Cited by 9
Matsubara, Y., Soldaini, L., Lind, E., & Moschitti, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.537 · Cited by 9
Quantization-aware and Tensor-compressed Training of Transformers for Natural Language Understanding
Yang, Z., Choudhary, S., Kunzmann, S., & Zhang, Z. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-2045 · Cited by 10
Multimodality Self-distillation for Fast Inference of Vision and Language Pretrained Models
Kong, J., Wang, J., Yu, L.-C., & Zhang, X. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3384060 · Cited by 8
ClipFormer : Key–Value Clipping of Transformers on Memristive Crossbars for Write Noise Mitigation
Bhattacharjee, A., Moitra, A., & Panda, P. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3435762 · Cited by 10
Yu, H., Su, W., Liu, L., Zhang, J., Cai, C., Xu, C., Quan, H., & Xie, Y. (2024) · Displays · doi:10.1016/j.displa.2024.102819 · Cited by 7
Self-citationHigh-performance network attack detection in unknown scenarios based on improved vertical model
Hou, S., Xiao, G., & Zhou, H. (2024) · Cluster Computing · doi:10.1007/s10586-024-04840-6 · Cited by 8
Hussain, M., Chen, C., Albouq, S. S., Shinan, K., Alanazi, F., Iqbal, M. W., & Ashraf, M. U. (2025) · PLoS ONE · doi:10.1371/journal.pone.0312867 · Cited by 9
Lightweight Cross-Lingual Sentence Representation Learning
Mao, Z., Gupta, P., Chu, C., Jaggi, M., & Kurohashi, S. (2021) · doi:10.18653/v1/2021.acl-long.226 · Cited by 7
Self-training vs Pre-trained Embeddings for Automatic Essay Scoring
Zhou, X., Yang, L., Fan, X., Ren, G., Yang, Y., & Lin, H. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88189-4_12 · Cited by 7
Self-citationEfficient Image and Sentence Matching
Huang, Y., Wang, Y., & Wang, L. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3178485 · Cited by 8
Tiny RNN Model with Certified Robustness for Text Classification
Qiang, Y., Kumar, S. T. S., Brocanelli, M., & Zhu, D. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892117 · Cited by 8
Text-based neural networks for question intent recognition
Trewhela, A., & Figueroa, A. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.105933 · Cited by 7
SPSQL: Step-by-step Parsing Based Framework for Text-to-SQL Generation
Shen, R., Sun, G., Shen, H., Li, Y., Jin, L., & Jiang, H. (2023) · International Conference on Machine Vision and Information Technology (CMVIT) · doi:10.1109/cmvit57620.2023.00030 · Cited by 10
Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection
Wang, C., Lu, Y., Mu, Y., Hu, Y., Xiao, T., & Zhu, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.464 · Cited by 10
Cost-effective Distillation of Large Language Models
Dasgupta, S., Cohn, T., & Baldwin, T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.463 · Cited by 10
A Fast Attention Network for Joint Intent Detection and Slot Filling on Edge Devices
Huang, L. P., Liang, S., Ye, F., & Gao, N. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3309272 · Cited by 13
Sentiment analysis with hotel customer reviews using FNet
Bhowmik, S., Sadik, R., Akanda, W., & Pavel, J. R. (2024) · Bulletin of Electrical Engineering and Informatics · doi:10.11591/eei.v13i2.6301 · Cited by 12
High Performance Natural Language Processing
Ilharco, G., Magalhaes, C. I., Turc, I., Dettmers, T., Ferreira, F., & Lee, K. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2020.emnlp-tutorials.4 · Cited by 7
Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering
Yang, S., & Seo, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.468 · Cited by 8
Magic Pyramid: Accelerating Inference with Early Exiting and Token PruningPreprint
He, X., Keivanloo, I., Xu, Y. T., He, X., Zeng, B., Rajagopalan, S., & Chilimbi, T. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.00230 · Cited by 12
PHILDER: Lightweight Framework for Intelligent Phishing Detection on Resource-Limited Devices
Bispo, G. D., de Andrade, C. A. B., Saiki, G. M., Borges, R. V., Serrano, A. L. M., Filho, G. P. R., & Gonçalves, V. P. (2025) · IEEE Access · doi:10.1109/access.2025.3592541 · Cited by 6
PIMPAL: Accelerating LLM Inference on Edge Devices via In-DRAM Arithmetic Lookup
Jang, Y., Cho, H., Ryu, Y., Kim, J., & Hong, S. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11133391 · Cited by 6
MTF-ARD: multi-teacher adversarial distillation for robust person re-identification
Li, Y., Ye, J., Zhang, H., & Wang, Q. (2025) · The Visual Computer · doi:10.1007/s00371-025-04150-x · Cited by 6
InFi: End-to-End Learning to Filter Input for Resource-Efficiency in Mobile-Centric Inference
Yuan, M., Zhang, L., He, F., Tong, X., Song, M.-H., Xu, Z., & Li, X. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3275981 · Cited by 6
Exploring Transformers as Compact, Data-efficient Language Models
Fields, C., & Kennington, C. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.35 · Cited by 6
Over-the-Air Computation Empowered Vertically Split Inference
Yang, P., Wen, D., Zeng, Q., Zhou, Y., Wang, T., Cai, H., & Shi, Y. (2024) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2024.3485678 · Cited by 6
No Fine-Tuning, No Cry: Robust SVD for Compressing Deep Networks
Tukan, M., Maalouf, A., Weksler, M., & Feldman, D. (2021) · Sensors · doi:10.3390/s21165599 · Cited by 19
RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation
Lu, P., Ghaddar, A., Rashid, A., Rezagholizadeh, M., Ghodsi, A., & Langlais, P. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.270 · Cited by 6
Meta-learning for compressed language model: A multiple choice question answering study
Yan, M., & Pan, Y. F. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.01.148 · Cited by 6
Analysis on Tweets Towards COVID-19 Pandemic: An Application of Text-Based Depression Detection
Kaseb, A., Galal, O., & Elreedy, D. (2022) · Novel Intelligent and Leading Emerging Sciences Conference (NILES) · doi:10.1109/niles56402.2022.9942363 · Cited by 6
Wang, H., Kang, X., & Ren, F. (2022) · Communications in computer and information science · doi:10.1007/978-981-19-7943-9_27 · Cited by 6
Bilaterally Slimmable Transformer for Elastic and Efficient Visual Question Answering
Zhou, Y., Jin, Z., Yu, J., Xu, M., Wang, H., & Jianping Fan, B. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3254205 · Cited by 6
Stochastically Pruning Large Language Models Using Sparsity Regularization and Compressive Sensing
Islam, M. M., & Alawad, M. (2023) · ACM Great Lakes Symposium on VLSI (GLSVLSI) · doi:10.1145/3583781.3590232 · Cited by 6
PET: Parameter-efficient Knowledge Distillation on Transformer
Jeon, H., Park, S., Kim, J.-G., & Kang, U. (2023) · PLoS ONE · doi:10.1371/journal.pone.0288060 · Cited by 6
Kqiku, L., & Reinhardt, D. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31357 · Cited by 7
An efficient system for grammatical error correction on mobile devices
Gothe, S. V., Dogra, S., Chandra, M. S. Y., Sanchi, C., & Raja, B. R. K. (2021) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc50631.2021.00034 · Cited by 6
Li, T., Rashid, A., Jafari, A., Sharma, P., Ghodsi, A., & Rezagholizadeh, M. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.65 · Cited by 12
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging
Lu, P., Kobyzev, I., Rezagholizadeh, M., Rashid, A., Ghodsi, A., & Langlais, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.363 · Cited by 8
Redwood: Using Collision Detection to Grow a Large-Scale Intent Classification Dataset
Larson, S., & Leach, K. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.45 · Cited by 7
Undivided Attention: Are Intermediate Layers Necessary for BERT?Preprint
Sridhar, S. N., & Sarah, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11881 · Cited by 8
Improving Computational Efficiency in Visual Reinforcement Learning via Stored EmbeddingsPreprint
Chen, L., Lee, K., Srinivas, A., & Abbeel, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.02886 · Cited by 6
MLPruning: A Multilevel Structured Pruning Framework for Transformer-based ModelsPreprint
Yao, Z., Ma, L., Shen, S., Keutzer, K., & Mahoney, M. W. (2021) · arXiv (Cornell University) · Cited by 6
Learning to Teach with Student FeedbackPreprint
Liu, Y., Sun, T., Qiu, X., & Huang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04641 · Cited by 6
GenAI at the Edge: Comprehensive Survey on Empowering Edge DevicesPreprint
Navardi, M., Aalishah, R., Fu, Y., Lin, Y., Li, H., Chen, Y., & Mohsenin, T. (2025) · Qeios · doi:10.32388/jeu3u0 · Cited by 6
Transformer-Based Fault Detection Using Pressure Signals for Hydraulic Pumps
Kim, A. R., Kim, H. S., & Kim, S. Y. (2024) · IEEE Access · doi:10.1109/access.2024.3472750 · Cited by 8
Semantic and Precise Trigger Inversion: Detecting Backdoored Language Models
Xie, C., He, J., Yang, Y., Guo, S., Zhang, T., & Xiang, T. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3594037 · Cited by 6
Self-citationSimultaneously Training and Compressing Vision-and-Language Pre-Training Model
Qi, Q., Zhang, A., Liao, Y., Sun, W., Wang, Y., Li, X., & Liu, S. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3233258 · Cited by 6
NtNDet: Hardware Trojan detection based on pre-trained language models
Kuang, S., Quan, Z., Xie, G., Cai, X., Chen, X., & Li, K. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126666 · Cited by 7
Towards Semantics-Enhanced Pre-Training: Can Lexicon Definitions Help Learning Sentence Meanings?
Ren, X., Sun, X., Wang, H., & Liu, Q. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17619 · Cited by 6
A COVID-19 Search Engine (CO-SE) with Transformer-based architecture
Raza, S. (2022) · Healthcare Analytics · doi:10.1016/j.health.2022.100068 · Cited by 7
A novel multi-domain machine reading comprehension model with domain interference mitigation
Zhou, C., Wang, Z., He, S., Zhang, H., & Su, J. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.05.102 · Cited by 5
Video Event Extraction with Multi-View Interaction Knowledge Distillation
Wei, K., Du, R., Jin, L., Liu, J., Yin, J., Zhang, L., Liu, J., Liu, N., Zhang, J., & Guo, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29891 · Cited by 5
Panopoulos, I., Venieris, S. I., & Venieris, I. S. (2024) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3665868 · Cited by 6
TS ‐Mixer: A lightweight text representation model based on context awareness
Tang, H., Wang, Y., Zhang, Y., Dou, Q., & Lu, M. (2024) · Expert Systems · doi:10.1111/exsy.13732 · Cited by 5
Automatic Related Work Section Generation by Sentence Extraction and Reordering.
Deng, Z., Zeng, Z., Gu, W., Ji, J., & Bolin, H. (2021) · Cited by 5
Student-Teacher Oneness: A Storage-efficient approach that improves facial expression recognition
Zheng, Z., Rasmussen, C. E., & Peng, X. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00453 · Cited by 5
E-LANG: Energy-Based Joint Inferencing of Super and Swift Language Models
Akbari, M., Banitalebi-Dehkordi, A., & Zhang, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.359 · Cited by 8
PointSplit: Towards On-device 3D Object Detection with Heterogeneous Low-power Accelerators
Park, K., Choi, Y. R., Lee, I., & Kim, H. (2023) · ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) · doi:10.1145/3583120.3587045 · Cited by 5
Feature structure distillation with Centered Kernel Alignment in BERT transferring
Jung, H.-J., Kim, D., Na, S., & Kim, K. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120980 · Cited by 5
LittleBird: Efficient Faster & Longer Transformer for Question Answering
Lee, M., Han, K., & Shin, M. C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.352 · Cited by 7
EMS: Efficient and Effective Massively Multilingual Sentence Embedding Learning
Mao, Z., Chu, C., & Kurohashi, S. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3402064 · Cited by 8
Cahyawijaya, S. (2021) · arXiv.org · doi:10.14711/thesis-991012980421703412 · Cited by 10
Vis-TOP: Visual Transformer Overlay ProcessorPreprint
Hu, W., Xu, D. G., Fan, Z., Liu, F., & He, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10957 · Cited by 7
Li, T., Rashid, A., Jafari, A., Sharma, P., Ghodsi, A., & Rezagholizadeh, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05696 · Cited by 5
Wei, Q., Zhang, X., Liang, X., Guo, Z., Zhang, Y., & Chen, Z. (2025) · Digital Health · doi:10.1177/20552076241310060 · Cited by 4
Dynamic Model Merging With Mixture of Weights
Ye, H., Huang, C., Shen, M., Chen, T., Huang, Y., & Ouyang, W. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3544680 · Cited by 6
Lu, X., Zhao, Y., Qin, B., & Liu, T. (2025) · Electronics · doi:10.3390/electronics14214256 · Cited by 4
Guan, R., Liu, J., Jia, L., Zhao, H., Yao, S., Zhu, X., Man, K. L., Lim, E. G., Smith, J. S., & Yue, Y. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11246532 · Cited by 4
Light Pre-Trained Chinese Language Model for NLP Tasks
Li, J., Hu, H., Zhang, X., Li, M., Li, L., & Xu, L. J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60457-8_47 · Cited by 3
Distill-AER: Fine-Grained Address Entity Recognition from Spoken Dialogue via Knowledge Distillation
Wang, Y., Han, X., Zhou, F., Wang, Y., Deng, C., & Feng, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17120-8_50 · Cited by 4
Ko, J., Park, S., Jeong, M., Hong, S.-J., Ahn, E., Chang, D.-S., & Yun, S.-Y. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.12 · Cited by 5
Tiny Language Models Enriched with Multimodal Knowledge from Multiplex Networks
Fields, C., Natouf, O., McMains, A., Henry, C., & Kennington, C. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.3 · Cited by 4
Transformer-Based Model for Malicious URL Classification
Do, N. Q., Selamat, A., Lim, K. C., Krejcar, O., & Ghani, N. A. M. (2023) · IEEE International Conference on Computing (ICOCO) · doi:10.1109/icoco59262.2023.10397705 · Cited by 4
No Head Left Behind – Multi-Head Alignment Distillation for Transformers
Zhao, T., Singh, K. Y., Appalaraju, S., Tang, P., Mahadevan, V., Manmatha, R., & Wu, Y. N. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i7.28583 · Cited by 6
DP-FFN: Block-Based Dynamic Pooling for Accelerating Feed-Forward Layers in Transformers
Tang, J., Wang, S., Chen, S., & Kang, Y. (2024) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas58744.2024.10558119 · Cited by 4
Shakhadri, S. A. G., Kruthika, K. R., & Aralimatti, R. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-96231-8_32 · Cited by 13
Collaborative Distillation Strategies for Parameter-Efficient Language Model Deployment
Meng, X., Wu, Y., Tian, Y., Hu, X., Kang, T., & Du, J. (2025) · International Conference on Computer Information Science and Application Technology (CISAT) · doi:10.1109/cisat66811.2025.11181792 · Cited by 4
Kim, B., Seo, S., Han, S., Erdenee, E., & Chang, B. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.286 · Cited by 6
Li, W., Gao, S., ZHANG, J., & Guo, S.-X. (2021) · Scientific Programming · doi:10.1155/2021/5753693 · Cited by 4
Pay attention to the hidden semanteme
Tang, H., Liu, X., Wang, Y., Dou, Q., & Lu, M. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119076 · Cited by 4
Gradient-based Intra-attention Pruning on Pre-trained Language Models
Yang, Z., Cui, Y., Yao, X., & Wang, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.156 · Cited by 7
Mask Again: Masked Knowledge Distillation for Masked Video Modeling
Li, X., He, S., Wu, J., Yu, Y., Nie, L., & Zhang, M. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612129 · Cited by 5
Three Decades of Low Power: From Watts to Wisdom
Munir, M., Modi, S., Cooper, G., Kim, H., & Mărculescu, R. (2024) · IEEE Access · doi:10.1109/access.2024.3361484 · Cited by 4
Xing, X., Du, L., Wang, X., Zeng, X., Wang, Y., Zhang, Z., & Zhang, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i14.29542 · Cited by 5
Audio DistilBERT: A Distilled Audio BERT for Speech Representation Learning
Yu, F., Guo, J., Xi, W., Yang, Z., Jiang, R., & Zhang, C. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533328 · Cited by 4
Metadata
- Authors
- Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., & Zhou, D.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 2158-2170
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3034457371
- Cited by
- 757
- References
- 54
- Citations by year
- 2026: 94 · 2025: 134 · 2024: 98 · 2023: 128 · 2022: 119 · 2021: 140 · 2020: 43
- FWCI
- 41.07
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- MobileBERT, BERT, model compression, knowledge distillation, teacher-student learning, self-attention, feedforward neural networks, mobile devices, resource-constrained deployment, natural language inference, question answering, adhesives, SQuAD
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
