The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
FTRANS: energy-efficient acceleration of transformers using FPGA
Abstract
In natural language processing (NLP), the "Transformer" architecture was proposed as the first transduction model replying entirely on self-attention mechanisms without using sequence-aligned recurrent neural networks (RNNs) or convolution, and it achieved significant improvements for sequence to sequence tasks. The introduced intensive computation and storage of these pre-trained language representations has impeded their popularity into computation and memory constrained devices. The field-programmable gate array (FPGA) is widely used to accelerate deep learning algorithms for its high parallelism and low latency. However, the trained models are still too large to accommodate to an FPGA fabric. In this paper, we propose an efficient acceleration framework, Ftrans, for transformer-based large scale language representations. Our framework includes enhanced block-circulant matrix (BCM)-based weight representation to enable model compression on large-scale language representations at the algorithm level with few accuracy degradation, and an acceleration design at the architecture level. Experimental results show that our proposed framework significantly reduce the model size of NLP models by up to 16 times. Our FPGA design achieves 27.07× and 81 × improvement in performance and energy efficiency compared to CPU, and up to 8.80× improvement in energy efficiency compared to GPU.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The embedding layer has only one-third of the total parameters; hence, they store it off-chip.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [250]
In the citing paper
“The previous works on BCM only take the first row/column as the index vector but fail to represent the remaining rows/columns.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [250]
In the citing paper
“Their technique brings up to 16X reduction in model size and also improves performance and energy efficiency compared to CPU.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [250]
In the citing paper
“The encoder/decoder layers, which account for two-thirds of total parameters, are stored on-chip, which reduces off-chip communication.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [250]
In the citing paper
“Ftrans [250] technique proposes an improved BCM approach for compressing transformer models.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [250]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
(2017) · Cited by 11,625
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 3,710
5th International Conference on Learning Representations (ICLR 17)
Havrylov, S., & Titov, I. (2017) · Cited by 3,503
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-4009 · Cited by 2,475
Structured Matrices and Polynomials: Unified Superfast Algorithms
Pan, V. Y. (2001) · Cited by 329
Ding, C., Liao, S., Wang, Y., Li, Z., Liu, N., Zhuo, Y., Wang, C., Qian, X., Bai, Y., Yuan, G., et al. (2017) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3123939.3124552 · Cited by 259
Self-citationWang, S., Li, Z., Ding, C., Yuan, B., Qiu, Q., Wang, Y., & Liang, Y. E. (2018) · ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) · doi:10.1145/3174243.3174253 · Cited by 221
Self-citation
Research works printed but not linked: 5
- Kyunghyun Cho , Bart Van Merriënboer , Dzmitry Bahdanau, and Yoshua Bengio. 2014 . On the properties of neural machine translation: encoder-decoder approaches. arXiv preprint arXiv:1409.1259. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: encoder-decoder approaches. arXiv preprint arXiv:1409.1259. · arXiv
- Caiwen Ding Siyu Liao Yanzhi Wang Zhe Li etal 2017. CirCNN: Accelerating and Compressing Deep Neural Networks using Block-circulant Weight Matrices. In MICRO. ACM 395--408. Caiwen Ding Siyu Liao Yanzhi Wang Zhe Li et al. 2017. CirCNN: Accelerating and Compressing Deep Neural Networks using Block-circulant Weight Matrices. In MICRO. ACM 395--408.
- Yoon Kim Carl Denton Luong Hoang and Alexander M Rush. 2017. Structured attention networks. arXiv preprint arXiv:1702.00887. Yoon Kim Carl Denton Luong Hoang and Alexander M Rush. 2017. Structured attention networks. arXiv preprint arXiv:1702.00887. · arXiv
- Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems 5998--6008. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems 5998--6008.
- Liang Zhao , Siyu Liao , Yanzhi Wang , Zhe Li , Jian Tang , and Bo Yuan . 2017 . Theoretical properties for neural networks with weight matrices of low displacement rank . In International Conference on Machine Learning, 4082--4090 . Liang Zhao, Siyu Liao, Yanzhi Wang, Zhe Li, Jian Tang, and Bo Yuan. 2017. Theoretical properties for neural networks with weight matrices of low displacement rank. In International Conference on Machine Learning, 4082--4090.
Cited by
Sorted by citations
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Transformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,523
Sanger: A Co-Design Framework for Enabling Sparse Attention using Reconfigurable Architecture
Lu, L., Jin, Y., Bi, H., Luo, Z., Li, P., Wang, T., & Liang, Y. E. (2021) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3466752.3480125 · Cited by 249
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154
ViA: A Novel Vision-Transformer Accelerator Based on FPGA
Wang, T., Gong, L., Wang, C., Yang, Y., Gao, Y., Zhou, X., & Chen, H. (2022) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2022.3197489 · Cited by 150
Metadata
- Authors
- Li, B., Pandey, S., Fang, H., Lyv, Y., Li, J., Chen, J., Xie, M., Wan, L., Liu, H., & Ding, C.
- Affiliations
- University of Connecticut (US)
- Stevens Institute of Technology (US)
- Syracuse University (US)
- Microsoft (United States) (US)
- Oak Ridge National Laboratory (US)
- The University of Texas at San Antonio (US)
- ORCID
- Venue
- Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design
- Publisher
- ACM
- Published
- 2020-08-10
- Pages
- 175-180
- OpenAlex
- W3047848469
- Cited by
- 206
- References
- 14
- Citations by year
- 2026: 23 · 2025: 54 · 2024: 55 · 2023: 38 · 2022: 25 · 2021: 11
- FWCI
- 13.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium on Low Power Electronics and Design (ISLPED) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Electrical and Electronic Engineering › VLSI and FPGA Design Techniques
- Keywords
- transformer models, FPGA acceleration, model compression, block circulant matrices, self-attention, energy efficiency, hardware architecture design, deep learning acceleration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
