An Efficient Hardware Accelerator for Sparse Transformer Neural Networks
Abstract
Transformers have been an indispensable staple in deep learning. However, it is challenging to realize efficient deployment for Transformer-based model due to their substantial computation and memory demands. To address this issue, we present an efficient sparse Transformer accelerator on FPGA, namely STA, by exploiting N:M fine-grained structured sparsity. Our design features not only a unified computing engine capable of performing both sparse and dense matrix multiplications with high computational efficiency, but also a scalable softmax module eliminating the latency from intermediate off-chip data communication. Experimental results show that our implementation achieves the lowest latency compared to CPU, GPU, and prior FPGA-based accelerators. Moreover, compared with the state-of the-art FPGA-based accelerators, it can achieve up to $12.28\times$ and $51.00\times$ improvement on energy efficiency and MAC efficiency, respectively.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A transformer having N:M sparsity requires both sparse-dense and dense-dense MatMul.
“A transformer having N:M sparsity requires both sparse-dense and dense-dense MatMul.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [89]
For a fixed sparsity ratio, setting N to 1 provides comparable accuracy as N=2.
“They note that for a fixed sparsity ratio (i.e., fixed (MN)/N), setting N to 1 provides comparable accuracy as N=2, and hence, they set N=1.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [89]
On using N:M of 1:8, latency reduces further.
“On using N:M of 1:8, the latency reduces further.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [89]
For sparse-dense MatMul, trivial computations are avoided, improving hardware efficiency and performance.
“Clearly, for sparse-dense MatMul, trivial computations are avoided, improving hardware efficiency and performance.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [89]
Compared to Lu et al.'s design with 4096 MAC units, Fang et al.'s design has comparable inference latency and much higher throughput per MAC unit.
“Compared to Lu et al. [119], which uses 4096 MAC units, their design has comparable inference latency and much higher throughput per MAC unit.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [89]
Accuracy loss remains negligible even with 75% sparsity.
“The accuracy loss remains negligible even with 75% sparsity; therefore, they set N:M value to 1:4.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [89]
Cites
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 1× · see citing works
“Dino [18], a tiny vision Transformer, can be the backbone for many computer vision tasks.”
BackgroundFTRANS: energy-efficient acceleration of transformers using FPGA
Li, B., Pandey, S., Fang, H., Lyv, Y., Li, J., Chen, J., Xie, M., Wan, L., Liu, H., & Ding, C. (2020) · Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design · doi:10.1145/3370748.3406567 · Cited by 206 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154 · see citing works
- Authors
- Fang, C., Guo, S., Wu, W., Lin, J., Wang, Z., Hsu, M. K., & Liu, L.
- Affiliations
- Nanjing University (CN)
- Kuaishou (China) (CN)
- Venue
- 2022 IEEE International Symposium on Circuits and Systems (ISCAS)
- Publisher
- IEEE
- Published
- 2022-05-28
- Pages
- 2670-2674
- ISSN
- 0271-4302 (ISSN-L)
- OpenAlex
- W4312569226
- Cited by
- 30
- References
- 3
- Citations by year
- 2026: 5 · 2025: 11 · 2024: 10 · 2023: 3 · 2022: 1
- FWCI
- 2.27
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Symposium on Circuits and Systems proceedings (publishedVersion)
- Funders
- National Natural Science Foundation of China
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
- Physical Sciences › Engineering › Electrical and Electronic Engineering › VLSI and FPGA Design Techniques
- Keywords
- sparse transformer, FPGA accelerator, structured sparsity, sparse matrix multiplication, dense matrix multiplication, energy efficiency, hardware acceleration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
