HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
Open accessAbstract
Transformers are ubiquitous in Natural Language Processing (NLP) tasks, but they are difficult to be deployed on hardware due to the intensive computation.To enable low-latency inference on resource-constrained hardware platforms, we propose to design Hardware-Aware Transformers (HAT) with neural architecture search.We first construct a large design space with arbitrary encoder-decoder attention and heterogeneous layers.Then we train a Super-Transformer that covers all candidates in the design space, and efficiently produces many SubTransformers with weight sharing.Finally, we perform an evolutionary search with a hardware latency constraint to find a specialized SubTransformer dedicated to run fast on the target hardware.Extensive experiments on four machine translation tasks demonstrate that HAT can discover efficient models for different hardware (CPU, GPU, IoT device).When running WMT'14 translation task on Raspberry Pi-4, HAT can achieve 3× speedup, 3.7× smaller size over baseline Transformer; 2.7× speedup, 3.6× smaller size over Evolved Transformer with 12,041× less search cost and no performance loss.HAT is open-sourced.Elastic Layer Num in Encoder Elastic Head Num (Self Attention) Elastic Hidden Dim in FFN Encoder Layer 2 Encoder Layer m Elastic Embedding Dim Elastic Head Num (Self Attention) Elastic Hidden Dim in FFN Elastic Embedding Dim Elastic Head Num (En-Decoder Attention) Decoder Layer n Elastic Layer Num in Decoder Arbitrary Encoder-Decoder Attention concat ❶ Train a SuperTransformer by uniformly sampling SubTransformers with weight sharing 12/3/
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
HAT developed a methodology to search transformer models which have better validation metrics than the original transformer while having lower latency on CPU, GPU and mobile platforms.
“For instance, Hardware-aware Transformers (HAT) [53] developed a methodology to search transformer models which have better validation metrics than original transformer [2] while having lower latency on CPU, GPU and mobile platforms.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [53]
HAT showed that the NLP transformer specialized for GPU runs faster on GPU than CPU and vice versa for similar validation accuracy.
“For instance, Hardware-aware Transformers (HAT) [53] showed that the NLP transformer specialized for GPU runs faster on GPU than CPU and vice versa for similar validation accuracy.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [53]
HAT's searched subtransformers reveal that GPU platforms prefer shallow models with wide layers, while ARM CPU picks deep models with thin layers to obtain optimal hardware performance.
“The searched subtransformers reveal that GPU platforms prefer shallow models with wide layers, while ARM CPU picks deep models with thin layers to obtain optimal hardware performance.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [53]
HAT's searched model for Raspberry Pi-4 embedded device on the WMT'14 translation task achieves a 3× speedup with 3.7× fewer trainable parameters over the baseline vanilla transformer.
“The searched model for Raspberry Pi-4 embedded device on the WMT'14 translation task achieves a 3 × speedup with 3.7 × fewer trainable parameters over the baseline vanilla transformer.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [53]
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
MnasNet: Platform-Aware Neural Architecture Search for Mobile
Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., & Le, Q. V. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00293 · Cited by 3,416 · see citing works
Convolutional Sequence to Sequence Learning
Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850 · cited 1× · see citing works
“…En-De, we apply 32K source-target BPE vocabulary, train on WMT’16, validate on newstest2013 and test on newstest2014, replicating Wu et al. (2019b); For WMT’14 En-Fr, we use 40K source-target BPE vocabulary, validate on newstest2012&2013, and test on newstest2014, replicating Gehring et al. (2017).”
MethodSequence-Level Knowledge Distillation
Kim, Y., & Rush, A. M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1139 · Cited by 836 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Efficient transformers: A survey
Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2023) · ACM Computing Surveys · doi:10.1145/3530811 · Cited by 1,267 · see citing works
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3226481 · Cited by 436 · see citing works
“Hardware-aware transformer designs using NAS techniques for edge-compatible and low-latency inference are presented in [68].”
MethodAutoFormer: Searching Transformers for Visual Recognition
Chen, M., Peng, H., Fu, J., & Ling, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01205 · Cited by 323 · see citing works
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154 · see citing works
DynaBERT: Dynamic BERT with Adaptive Width and Depth
Hou, L., Huang, Z., Shang, L., Jiang, X., Chen, X., & Liu, Q. (2020) · arXiv · doi:10.48550/arxiv.2004.04037 · Cited by 143 · see citing works
Neural Architecture Search Survey: A Hardware Perspective
Chitty-Venkata, K. T., & Somani, A. K. (2023) · ACM Computing Surveys · doi:10.1145/3524500 · Cited by 106 · see citing works
NAS4RRAM: neural network architecture search for inference on RRAM-based accelerators
Yuan, Z., Liu, J., Li, X., Yan, L., Chen, H., Wu, B., Yang, Y., & Sun, G. (2021) · Science China Information Sciences · doi:10.1007/s11432-020-3245-7 · Cited by 30 · see citing works
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models
Yin, Y., Chen, C., Shang, L., Jiang, X., Chen, X., & Liu, Q. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.400 · Cited by 24 · see citing works
Real-time style transfer with efficient vision transformers
Benmeziane, H., Ouarnoughi, H., Maghraoui, K. E., & Niar, S. (2022) · Proceedings of the 5th International Workshop on Edge Systems, Analytics and Networking · doi:10.1145/3517206.3526271 · Cited by 5 · see citing works
- Authors
- Wang, H., Wu, Z., Liu, Z., Cai, H., Zhu, L., Gan, C., & Han, S.
- Affiliations
- Massachusetts Institute of Technology (US)
- IBM (United States) (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 7675-7688
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3035251378
- Cited by
- 227
- References
- 4
- Citations by year
- 2026: 11 · 2025: 22 · 2024: 30 · 2023: 64 · 2022: 38 · 2021: 48 · 2020: 14
- FWCI
- 12.65
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1943349)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- neural architecture search, low-latency inference, resource-constrained devices, evolutionary search, parameter sharing, heterogeneous layer, encoder-decoder attention, model compression, Raspberry Pi deployment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
