The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
FTRANS: energy-efficient acceleration of transformers using FPGA
Abstract
In natural language processing (NLP), the "Transformer" architecture was proposed as the first transduction model replying entirely on self-attention mechanisms without using sequence-aligned recurrent neural networks (RNNs) or convolution, and it achieved significant improvements for sequence to sequence tasks. The introduced intensive computation and storage of these pre-trained language representations has impeded their popularity into computation and memory constrained devices. The field-programmable gate array (FPGA) is widely used to accelerate deep learning algorithms for its high parallelism and low latency. However, the trained models are still too large to accommodate to an FPGA fabric. In this paper, we propose an efficient acceleration framework, Ftrans, for transformer-based large scale language representations. Our framework includes enhanced block-circulant matrix (BCM)-based weight representation to enable model compression on large-scale language representations at the algorithm level with few accuracy degradation, and an acceleration design at the architecture level. Experimental results show that our proposed framework significantly reduce the model size of NLP models by up to 16 times. Our FPGA design achieves 27.07× and 81 × improvement in performance and energy efficiency compared to CPU, and up to 8.80× improvement in energy efficiency compared to GPU.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The embedding layer has only one-third of the total parameters; hence, they store it off-chip.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [250]
In the citing paper
“The previous works on BCM only take the first row/column as the index vector but fail to represent the remaining rows/columns.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [250]
In the citing paper
“Their technique brings up to 16X reduction in model size and also improves performance and energy efficiency compared to CPU.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [250]
In the citing paper
“The encoder/decoder layers, which account for two-thirds of total parameters, are stored on-chip, which reduces off-chip communication.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [250]
In the citing paper
“Ftrans [250] technique proposes an improved BCM approach for compressing transformer models.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [250]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
(2017) · Cited by 11,625
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 3,710
5th International Conference on Learning Representations (ICLR 17)
Havrylov, S., & Titov, I. (2017) · Cited by 3,503
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-4009 · Cited by 2,475
Structured Matrices and Polynomials: Unified Superfast Algorithms
Pan, V. Y. (2001) · Cited by 329
Ding, C., Liao, S., Wang, Y., Li, Z., Liu, N., Zhuo, Y., Wang, C., Qian, X., Bai, Y., Yuan, G., et al. (2017) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3123939.3124552 · Cited by 259
Self-citationWang, S., Li, Z., Ding, C., Yuan, B., Qiu, Q., Wang, Y., & Liang, Y. E. (2018) · ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) · doi:10.1145/3174243.3174253 · Cited by 221
Self-citation
Research works printed but not linked: 5
- Kyunghyun Cho , Bart Van Merriënboer , Dzmitry Bahdanau, and Yoshua Bengio. 2014 . On the properties of neural machine translation: encoder-decoder approaches. arXiv preprint arXiv:1409.1259. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: encoder-decoder approaches. arXiv preprint arXiv:1409.1259. · arXiv
- Caiwen Ding Siyu Liao Yanzhi Wang Zhe Li etal 2017. CirCNN: Accelerating and Compressing Deep Neural Networks using Block-circulant Weight Matrices. In MICRO. ACM 395--408. Caiwen Ding Siyu Liao Yanzhi Wang Zhe Li et al. 2017. CirCNN: Accelerating and Compressing Deep Neural Networks using Block-circulant Weight Matrices. In MICRO. ACM 395--408.
- Yoon Kim Carl Denton Luong Hoang and Alexander M Rush. 2017. Structured attention networks. arXiv preprint arXiv:1702.00887. Yoon Kim Carl Denton Luong Hoang and Alexander M Rush. 2017. Structured attention networks. arXiv preprint arXiv:1702.00887. · arXiv
- Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems 5998--6008. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems 5998--6008.
- Liang Zhao , Siyu Liao , Yanzhi Wang , Zhe Li , Jian Tang , and Bo Yuan . 2017 . Theoretical properties for neural networks with weight matrices of low displacement rank . In International Conference on Machine Learning, 4082--4090 . Liang Zhao, Siyu Liao, Yanzhi Wang, Zhe Li, Jian Tang, and Bo Yuan. 2017. Theoretical properties for neural networks with weight matrices of low displacement rank. In International Conference on Machine Learning, 4082--4090.
Cited by
Sorted by citations
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Transformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,523
Sanger: A Co-Design Framework for Enabling Sparse Attention using Reconfigurable Architecture
Lu, L., Jin, Y., Bi, H., Luo, Z., Li, P., Wang, T., & Liang, Y. E. (2021) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3466752.3480125 · Cited by 249
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154
ViA: A Novel Vision-Transformer Accelerator Based on FPGA
Wang, T., Gong, L., Wang, C., Yang, Y., Gao, Y., Zhou, X., & Chen, H. (2022) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2022.3197489 · Cited by 150
HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformers
Dong, P., Sun, M., Lu, A., Xie, Y., Liu, K., Kong, Z., Meng, X., Li, Z., Lin, X., Fang, Z., et al. (2023) · 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071047 · Cited by 128
Accelerating Transformer-based Deep Learning Models on FPGAs using Column Balanced Block Pruning
Peng, H., Huang, S., Geng, T., Li, A., Jiang, W., Liu, H., Wang, S., & Ding, C. (2021) · 2021 22nd International Symposium on Quality Electronic Design (ISQED) · doi:10.1109/isqed51717.2021.9424344 · Cited by 105
Self-citationAn Algorithm–Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers
Fang, C., Zhou, A., & Wang, Z. (2022) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2022.3197282 · Cited by 94
Li, Z., Sun, M., Lu, A., Ma, H., Yuan, G., Xie, Y., Tang, H., Li, Y., Leeser, M., Wang, Z., et al. (2022) · 2022 32nd International Conference on Field-Programmable Logic and Applications (FPL) · doi:10.1109/fpl57034.2022.00027 · Cited by 85
Understanding the Potential of FPGA-based Spatial Acceleration for Large Language Model Inference
Chen, H., Zhang, J., Du, Y., Xiang, S., Yue, Z., Zhang, N., Cai, Y., & Zhang, Z. (2024) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3656177 · Cited by 98
Huang, M., Luo, J., Ding, C., Wei, Z., Huang, S., & Yu, H. (2023) · IEEE Transactions on Circuits and Systems I: Regular Papers · doi:10.1109/tcsi.2023.3312775 · Cited by 77
Self-citationFPGA-based Deep Learning Inference Accelerators: Where Are We Standing?
Nechi, A., Groth, L., Mulhem, S., Merchant, F., Buchty, R., & Bereković, M. (2023) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3613963 · Cited by 94
ViTA: A Vision Transformer Inference Accelerator for Edge Applications
Nag, S., Datta, G., Kundu, S., Chandrachoodan, N., & Beerel, P. A. (2023) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas46773.2023.10181988 · Cited by 76
Adaptable butterfly accelerator for attention-based NNs via hardware and algorithm co-design
Fan, H., Chau, T., Venieris, S. I., Lee, R., Kouris, A., Luk, W., Lane, N. D., & Abdelfattah, M. S. (2022) · 2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro56248.2022.00050 · Cited by 66
EdgeLLM: A Highly Efficient CPU-FPGA Heterogeneous Edge Accelerator for Large Language Models
Huang, M., Shen, A., Li, K., Peng, H., Li, B., Su, Y., & Yu, H. (2025) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2025.3546256 · Cited by 84
Self-citationAccelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimization
Qi, P., Sha, E. H. ‐., Zhuge, Q., Peng, H., Huang, S., Kong, Z., Song, Y., & Li, B. (2021) · International Conference on Computer Aided Design · doi:10.1109/iccad51958.2021.9643586 · Cited by 56
Self-citationAccelTran: A Sparsity-Aware Accelerator for Dynamic Inference With Transformers
Tuli, S., & Jha, N. K. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3273992 · Cited by 60
Algorithm-hardware Co-design of Attention Mechanism on FPGA Devices
Zhang, X., Wu, Y., Zhou, P., Tang, X., & Hu, J. (2021) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3477002 · Cited by 52
Sarkar, R., Liang, H., Fan, Z., Wang, Z., & Hao, C. (2023) · 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD) · doi:10.1109/iccad57390.2023.10323651 · Cited by 64
A Survey on Hardware Accelerators for Large Language Models
Kachris, C. (2025) · Applied Sciences · doi:10.3390/app15020586 · Cited by 83
Zhao, Z., Cao, R., Un, K.-F., Yu, W.-H., Mak, P., & Martins, R. P. (2022) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2022.3196055 · Cited by 44
Review of neural network model acceleration techniques based on FPGA platforms
Liu, F., Li, H., Hu, W., & He, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128511 · Cited by 41
SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers
Marchisio, A., Dura, D., Capra, M., Martina, M., Masera, G., & Shafique, M. (2023) · 2023 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191521 · Cited by 39
Qi, P., Song, Y., Peng, H., Huang, S., Zhuge, Q., & Sha, E. H.-M. (2021) · Proceedings of the 2021 Great Lakes Symposium on VLSI · doi:10.1145/3453688.3461739 · Cited by 37
MECLA: Memory-Compute-Efficient LLM Accelerator with Scaling Sub-matrix Partition
Qin, Y., Wang, Y., Zhao, Z., Yang, X., Zhou, Y., Wei, S., Hu, Y., & Yin, S. (2024) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca59077.2024.00079 · Cited by 36
SOFA: A Compute-Memory Optimized Sparsity Accelerator via Cross-Stage Coordinated Tiling
Wang, H., Fang, J., Tang, X., Yue, Z., Li, J., Qin, Y., Guan, S., Yang, Q., Wang, Y., Li, C., et al. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00093 · Cited by 38
Self-citationMaxEVA: Maximizing the Efficiency of Matrix Multiplication on Versal AI Engine
Taka, E., Arora, A., Wu, K., & Marculescu, D. (2023) · International Conference on Field-Programmable Technology (FPT) · doi:10.1109/icfpt59805.2023.00016 · Cited by 33
A Fast and Flexible FPGA-based Accelerator for Natural Language Processing Neural Networks
Hur, S., Na, S., Kwon, D., Kim, J., Boutros, A., Nurvitadhi, E., & Kim, J. (2022) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3564606 · Cited by 37
Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLM
Yu, Z., Liang, S., Ma, T., Cai, Y., Nan, Z., Huang, D., Song, X., Hao, Y., Zhang, J., Zhi, T., et al. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00108 · Cited by 44
An Efficient Hardware Accelerator for Sparse Transformer Neural Networks
Fang, C., Guo, S., Wu, W., Lin, J., Wang, Z., Hsu, M. K., & Liu, L. (2022) · 2022 IEEE International Symposium on Circuits and Systems (ISCAS) · doi:10.1109/iscas48785.2022.9937659 · Cited by 30
HPTA: A High Performance Transformer Accelerator Based on FPGA
Han, Y., & Liu, Q. (2023) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl60245.2023.00012 · Cited by 31
Fu, X., Ren, Q., Wu, H., Xiang, F., Luo, Q., Yue, J., Chen, Y., & Zhang, F. (2023) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2023.3315060 · Cited by 30
An End-to-end Multi-task Object Detection using Embedded GPU in Autonomous Driving
Zhou, S., Xie, M., Jin, Y., Miao, F., & Ding, C. (2021) · International Symposium on Quality Electronic Design (ISQED) · doi:10.1109/isqed51717.2021.9424308 · Cited by 28
Self-citationSWAT: An Efficient Swin Transformer Accelerator Based on FPGA
Dong, Q.-W., Xie, X., & Wang, Z.-F. (2024) · Asia and South Pacific Design Automation Conference · doi:10.1109/asp-dac58780.2024.10473931 · Cited by 20
Row-wise Accelerator for Vision Transformer
Wang, H.-Y., & Chang, T.-S. (2022) · International Conference on Artificial Intelligence Circuits and Systems · doi:10.1109/aicas54282.2022.9869928 · Cited by 23
Hardware–Software Co-Design Enabling Static and Dynamic Sparse Attention Mechanisms
Zhao, J., Zeng, P., Shen, G., Chen, Q., & Guo, M. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3373592 · Cited by 25
Exploring HW/SW Co-Design for Video Analysis on CPU-FPGA Heterogeneous Systems
Zhang, X., Ma, Y., Xiong, J., Hwu, W., Kindratenko, V., & Chen, D. (2021) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2021.3093398 · Cited by 24
Jing, L., Dong, C., He, C., Shi, W., & Yin, H. (2022) · Remote Sensing · doi:10.3390/rs14235959 · Cited by 24
ASADI: Accelerating Sparse Attention Using Diagonal-based In-Situ Computing
Li, H., Li, Z., Bai, Z., & Mitra, T. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00065 · Cited by 23
Unified Accelerator for Attention and Convolution in Inference Based on FPGA
Li, T., Zhang, F., Fan, X., Shen, J., Guo, W., & Cao, W. (2023) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas46773.2023.10182145 · Cited by 22
SoftAct: A High-Precision Softmax Architecture for Transformers Supporting Nonlinear Functions
Fu, Y., Zhou, C., Huang, T., Han, E., He, Y., & Jiao, H. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3386779 · Cited by 21
CPSAA: Accelerating Sparse Attention Using Crossbar-Based Processing-In-Memory Architecture
Li, H., Jin, H., Zheng, L., Liao, X., Huang, Y., Liu, C., Xu, J., Duan, Z., Chen, D., & Gui, C. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3344524 · Cited by 21
PICACHU: Plug-In CGRA Handling Upcoming Nonlinear Operations in LLMs
Qin, J., Xia, T., Tan, C., Zhang, J. J., & Zhang, S. Q. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716013 · Cited by 26
EFA-Trans: An Efficient and Flexible Acceleration Architecture for Transformers
Yang, X., & Su, T. (2022) · Electronics · doi:10.3390/electronics11213550 · Cited by 18
A Survey on Sparsity Exploration in Transformer-Based Accelerators
Fuad, K. A. A., & Chen, L. (2023) · Electronics · doi:10.3390/electronics12102299 · Cited by 21
FET-OPU: A Flexible and Efficient FPGA-Based Overlay Processor for Transformer Networks
Bai, Y., Zhou, H., Zhao, K., Wang, H., Chen, J., Yu, J. W., & Wang, K. (2023) · doi:10.1109/iccad57390.2023.10323752 · Cited by 18
Self-citationFan, Z., Zhang, Q., Abillama, P., Shoouri, S., Lee, C. S. G., Blaauw, D. T., Kim, H.-S., & Sylvester, D. (2023) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3579371.3589040 · Cited by 16
Accelerating Transformer Neural Networks on FPGAs for High Energy Physics Experiments
Wojcicki, F., Que, Z., Tapper, A., & Luk, W. W. (2022) · International Conference on Field-Programmable Technology (FPT) · doi:10.1109/icfpt56656.2022.9974463 · Cited by 15
Wang, Z., Wang, G., Jiang, H., Xu, N., & He, G. (2023) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac56929.2023.10247678 · Cited by 17
Qin, Y., Lou, W., Wang, C., Gong, L., & Zhou, X. (2024) · ACM Great Lakes Symposium on VLSI (GLSVLSI) · doi:10.1145/3649476.3658810 · Cited by 15
Efficient memristor accelerator for transformer self-attention functionality
Bettayeb, M., Halawani, Y., Khan, M. U., Saleh, H. H., & Mohammad, B. S. (2024) · Scientific Reports · doi:10.1038/s41598-024-75021-z · Cited by 18
TRAC: Compilation-Based Design of Transformer Accelerators for FPGAs
Plagwitz, P., Hannig, F., & Teich, J. (2022) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl57034.2022.00015 · Cited by 15
LightMamba: Efficient Mamba Acceleration on FPGA with Quantization and Hardware Co-design
Wei, R., Xu, S., Zhong, L., Yang, Z., Guo, Q., Wang, Y., Wang, R., & Li, M. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10993079 · Cited by 17
Accelerating Transformer Networks through Recomposing Softmax Layers
Choi, J., Li, H., Kim, B., Hwang, S., & Ahn, J. H. (2022) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc55918.2022.00018 · Cited by 14
Mobile Transformer Accelerator Exploiting Various Line Sparsity and Tile-Based Dynamic Quantization
Kwon, E., Yoon, J., & Kang, S. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3347291 · Cited by 16
A Cost-Efficient FPGA-Based CNN-Transformer Using Neural ODE
Okubo, I., Sugiura, K., & Matsutani, H. (2024) · IEEE Access · doi:10.1109/access.2024.3480977 · Cited by 14
Sun, X., Zhang, Y., Wang, Q., Zou, X., Liu, Y., Zeng, Z., & Zhuang, H. (2025) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2025.105042 · Cited by 12
LTrans-OPU: A Low-Latency FPGA-Based Overlay Processor for Transformer Networks
Bai, Y., Zhou, H., Zhao, K., Zhang, M., Chen, J., Yu, J., & Wang, K. (2023) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl60245.2023.00048 · Cited by 12
Self-citationWang, H., Wang, Z., Yue, Z., Long, Y., Wei, T., Yang, J., Wang, Y., Li, C., Wei, S., Hu, Y., et al. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756037 · Cited by 11
Energy Efficient FPGA-Based Binary Transformer Accelerator for Edge Devices
Du, C., Ko, S., & Zhang, H. (2024) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas58744.2024.10558631 · Cited by 13
Binary Complex Neural Network Acceleration on FPGA : (Invited Paper)
Peng, H., Zhou, S., Weitze, S., Li, J., Islam, S., Geng, T., Li, A., Zhang, W., Song, M., Xie, M., et al. (2021) · IEEE International Conference on Application-specific Systems, Architectures and Processors (ASAP) · doi:10.1109/asap52443.2021.00021 · Cited by 14
Self-citationGRAEBO: FPGA General Routing Architecture Exploration via Bayesian Optimization
Zheng, S., Qian, J., Zhou, H., & Wang, L. (2022) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl57034.2022.00050 · Cited by 11
Hardware-friendly compression and hardware acceleration for transformer: A survey
Huang, S., Tang, E., Li, S., Ping, X., & Chen, R. (2022) · Electronic Research Archive · doi:10.3934/era.2022192 · Cited by 14
Inference Optimization of Foundation Models on AI Accelerators
Park, Y., Budhathoki, K., Chen, L., Kübler, J. M., Huang, J., Kleindeßner, M., Huan, J., Cevher, V., Wang, Y., & Karypis, G. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671465 · Cited by 16
In-Storage Acceleration of Retrieval Augmented Generation as a Service
Mahapatra, R., Santhanam, H., Priebe, C., Xu, H., & Esmaeilzadeh, H. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731032 · Cited by 13
Rizk, R., Rizk, D., Rizk, F., Kumar, A., & Bayoumi, M. (2022) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas48785.2022.9937531 · Cited by 10
A Self-Attention Network for Deep JSCCM: The Design and FPGA Implementation
Fujimaki, S., Inoue, Y., Hisano, D., Maruta, K., Nakayama, Y., & Hara–Azumi, Y. (2022) · Globecom · doi:10.1109/globecom48099.2022.10001518 · Cited by 10
Calabash: Accelerating Attention Using a Systolic Array Chain on FPGAs
Luo, Z., Lu, L., Jin, Y., Jia, L., & Liang, Y. E. (2023) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl60245.2023.00041 · Cited by 11
BETA: Binarized Energy-Efficient Transformer Accelerator at the Edge
Ji, Y., Fang, C., & Wang, Z. (2024) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas58744.2024.10558636 · Cited by 10
Kwon, E., Song, H., Park, J., & Kang, S. (2023) · Design, Automation, and Test in Europe · doi:10.23919/date56975.2023.10137099 · Cited by 9
EdgeTran: Device-Aware Co-Search of Transformers for Efficient Inference on Mobile Edge Platforms
Tuli, S., & Jha, N. K. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3328287 · Cited by 16
COSA Plus: Enhanced Co-Operative Systolic Arrays for Attention Mechanism in Transformers
Wang, Z., Wang, G., & He, G. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3434447 · Cited by 9
ChatOPU: An FPGA-based Overlay Processor for Large Language Models with Unstructured Sparsity
Zhao, T., Lu, S., Wu, C.-L., & He, L. (2024) · doi:10.1145/3676536.3676761 · Cited by 9
Reconfigurable Stream Network Architecture
Wang, C. C., Zhang, X., Cong, J., & Hoe, J. C. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731088 · Cited by 9
Liao, X., Zhou, T., Zhang, L., Hu, X. Y., & Peng, Y. (2023) · Electronics · doi:10.3390/electronics12020267 · Cited by 11
TransFRU: Efficient Deployment of Transformers on FPGA with Full Resource Utilization
Wang, H., Bai, Y., Yu, J., & Wang, K. (2024) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac58780.2024.10473976 · Cited by 8
Zhang, C., You, Y., Wang, N., Park, J., & Zhang, L. (2025) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2025.3575272 · Cited by 13
Hardware Acceleration of Transformer Networks using FPGAs
Τζάνος, Γ., Kachris, C., & Soudris, D. (2022) · Panhellenic Conference on Electronics & Telecommunications (PACET) · doi:10.1109/pacet56979.2022.9976354 · Cited by 8
Wu, J., Song, M., Zhao, J., Gao, Y., Li, J., & So, H. K. (2025) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3714416 · Cited by 10
Self-citationHardware Accelerator Design for Sparse DNN Inference and Training: A Tutorial
Mao, W., Wang, M., Xie, X., Wu, X., & Wang, Z. (2023) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2023.3344681 · Cited by 9
ChipMind: LLMs for Agile Chip Design
Firouzi, F., Pan, D. Z., Gu, J., Farahani, B., Chaudhuri, J., Yin, Z., Ma, P., Domanski, P., & Chakrabarty, K. (2025) · IEEE VLSI Test Symposium (VTS) · doi:10.1109/vts65138.2025.11022936 · Cited by 7
Co-Designing Binarized Transformer and Hardware Accelerator for Efficient End-to-End Edge Deployment
Ji, Y., Fang, C., Ma, S., Shao, H., & Wang, Z. (2024) · doi:10.1145/3676536.3676733 · Cited by 7
Byun, W.-H., Woo, J., & Mukhopadhyay, S. (2025) · IEEE Access · doi:10.1109/access.2025.3563196 · Cited by 8
LAD: Efficient Accelerator for Generative Inference of LLM with Locality Aware Decoding
Wang, H., Li, Y., Xu, H., Wang, Y., Liu, L., Yang, J., & Han, Y. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00111 · Cited by 6
Hardware Accelerator for Transformer based End-to-End Automatic Speech Recognition System
Yamini, S., Mirishkar, G. S., Vuppala, A. K., & Purini, S. (2023) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw59300.2023.00027 · Cited by 6
A Hardware Design Framework for Computer Vision Models Based on Reconfigurable Devices
Fan, Z., Hu, W., Liu, F., Xu, D. G., Guo, H., He, Y., & Peng, M. (2023) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3635157 · Cited by 6
Jang, S., & Cho, Y.-B. (2024) · Electronics · doi:10.3390/electronics13030552 · Cited by 6
Energy Efficient FPGA-Based Accelerator for Dynamic Sparse Transformer
Li, Z., Lai, Y., & Zhang, H. (2024) · International Conference on Communications, Circuits and Systems · doi:10.1109/icccas62034.2024.10652850 · Cited by 7
Energy efficient spike transformer accelerator at the edge
Du, C., Wen, Q., Wei, Z., & Zhang, H. (2024) · Intelligent Marine Technology and Systems · doi:10.1007/s44295-024-00040-5 · Cited by 7
Neural architecture search for adversarial robustness via learnable pruning
Li, Y., Zhao, P., Ding, R., Zhou, T. H., Fei, Y., Xu, X., & Lin, X. (2024) · Frontiers in High Performance Computing · doi:10.3389/fhpcp.2024.1301384 · Cited by 9
ProTEA: Programmable Transformer Encoder Acceleration on FPGA
Kabir, E., Bakos, J. D., Andrews, D. A., & Huang, M. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/scw63240.2024.00074 · Cited by 6
Dahal, S., Dhingra, P., Thapa, K. K., Pande, P. P., & Kalyanaraman, A. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3522781 · Cited by 7
Fan, Z., Huang, M., Zhang, X., Liu, R., Lyu, X., Duan, T., Bu, Z., & Liang, J. (2023) · Applied Sciences · doi:10.3390/app132212184 · Cited by 5
Song, H., Xu, L., Wang, Y., Wu, X., Wang, M., & Wang, Z. (2024) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas58744.2024.10557837 · Cited by 5
An FPGA-based Multi-Core Overlay Processor for Transformer-based Models
Lu, S., Zhao, T., Zhang, R., Lin, T.-J., Wu, C.-L., & He, L. (2024) · International Symposium of Electronics Design Automation (ISEDA) · doi:10.1109/iseda62518.2024.10617729 · Cited by 5
Wang, X., Miao, S., Qu, P., & Zhang, Y. (2024) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl64840.2024.00054 · Cited by 4
Low latency transformer inference on FPGAs for physics applications with hls4ml
Jiang, Z., Yin, D., Chen, Y.-H., Khoda, E. E., Hauck, S., Hsu, S.-C., Govorkova, E., Harris, P. C., Lončar, V., & Moreno, E. A. (2025) · Journal of Instrumentation · doi:10.1088/1748-0221/20/04/p04014 · Cited by 5
Enabling Energy-Efficient Inference for Self-Attention Mechanisms in Neural Networks
Chen, Q., Sun, C., Lu, Z., & Gao, C. (2022) · IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) · doi:10.1109/aicas54282.2022.9869924 · Cited by 6
Real block-circulant matrices and DCT-DST algorithm for transformer neural network
Asriani, E., Muchtadi-Alamsyah, I., & Purwarianti, A. (2023) · Frontiers in Applied Mathematics and Statistics · doi:10.3389/fams.2023.1260187 · Cited by 5
An FPGA-Based Efficient Streaming Vector Processing Engine for Transformer-Based Models
He, Z., Zhao, T., Miao, S., Wu, C., & He, L. (2024) · International Symposium of Electronics Design Automation (ISEDA) · doi:10.1109/iseda62518.2024.10617499 · Cited by 5
Metadata
- Authors
- Li, B., Pandey, S., Fang, H., Lyv, Y., Li, J., Chen, J., Xie, M., Wan, L., Liu, H., & Ding, C.
- Affiliations
- University of Connecticut (US)
- Stevens Institute of Technology (US)
- Syracuse University (US)
- Microsoft (United States) (US)
- Oak Ridge National Laboratory (US)
- The University of Texas at San Antonio (US)
- ORCID
- Venue
- Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design
- Publisher
- ACM
- Published
- 2020-08-10
- Pages
- 175-180
- OpenAlex
- W3047848469
- Cited by
- 206
- References
- 14
- Citations by year
- 2026: 23 · 2025: 54 · 2024: 55 · 2023: 38 · 2022: 25 · 2021: 11
- FWCI
- 13.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium on Low Power Electronics and Design (ISLPED) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Electrical and Electronic Engineering › VLSI and FPGA Design Techniques
- Keywords
- transformer models, FPGA acceleration, model compression, block circulant matrices, self-attention, energy efficiency, hardware architecture design, deep learning acceleration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
