FTRANS: energy-efficient acceleration of transformers using FPGA
Abstract
In natural language processing (NLP), the "Transformer" architecture was proposed as the first transduction model replying entirely on self-attention mechanisms without using sequence-aligned recurrent neural networks (RNNs) or convolution, and it achieved significant improvements for sequence to sequence tasks. The introduced intensive computation and storage of these pre-trained language representations has impeded their popularity into computation and memory constrained devices. The field-programmable gate array (FPGA) is widely used to accelerate deep learning algorithms for its high parallelism and low latency. However, the trained models are still too large to accommodate to an FPGA fabric. In this paper, we propose an efficient acceleration framework, Ftrans, for transformer-based large scale language representations. Our framework includes enhanced block-circulant matrix (BCM)-based weight representation to enable model compression on large-scale language representations at the algorithm level with few accuracy degradation, and an acceleration design at the architecture level. Experimental results show that our proposed framework significantly reduce the model size of NLP models by up to 16 times. Our FPGA design achieves 27.07× and 81 × improvement in performance and energy efficiency compared to CPU, and up to 8.80× improvement in energy efficiency compared to GPU.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Ftrans technique stores the embedding layer off-chip because it has only one-third of the total parameters.
“The embedding layer has only one-third of the total parameters; hence, they store it off-chip.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [250]
Previous works on block-circulant matrices only take the first row/column as the index vector but fail to represent the remaining rows/columns.
“The previous works on BCM only take the first row/column as the index vector but fail to represent the remaining rows/columns.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [250]
The Ftrans technique brings up to 16X reduction in model size and improves performance and energy efficiency compared to CPU.
“Their technique brings up to 16X reduction in model size and also improves performance and energy efficiency compared to CPU.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [250]
The Ftrans technique stores encoder/decoder layers on-chip, which account for two-thirds of total parameters, reducing off-chip communication.
“The encoder/decoder layers, which account for two-thirds of total parameters, are stored on-chip, which reduces off-chip communication.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [250]
The Ftrans technique proposes an improved block-circulant matrix approach for compressing transformer models.
“Ftrans [250] technique proposes an improved BCM approach for compressing transformer models.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [250]
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
(2017) · Cited by 11,625 · see citing works
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 3,710 · see citing works
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-4009 · Cited by 3,538 · see citing works
5th International Conference on Learning Representations (ICLR 17)
Havrylov, S., & Titov, I. (2017) · Cited by 3,503 · see citing works
Structured Matrices and Polynomials: Unified Superfast Algorithms
Pan, V. Y. (2001) · Cited by 329 · see citing works
Ding, C., Liao, S., Wang, Y., Li, Z., Liu, N., Zhuo, Y., Wang, C., Qian, X., Bai, Y., Yuan, G., et al. (2017) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3123939.3124552 · Cited by 260 · see citing works
Self-citationWang, S., Li, Z., Ding, C., Yuan, B., Qiu, Q., Wang, Y., & Liang, Y. E. (2018) · ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) · doi:10.1145/3174243.3174253 · Cited by 221 · see citing works
Self-citationMathematics of the Discrete Fourier Transform (DFT) with Audio Applications
Smith, & Olaniyan, J. (2007) · Cited by 218 · see citing works
Cooley-Tukey FFT on the Connection Machine
Johnsson, S. L., & Krawitz, R. L. (1992) · Parallel Computing · doi:10.1016/0167-8191(92)90066-g · Cited by 56 · see citing works
Towards Neural Machine Translation with Latent Tree Attention
Bradbury, J., & Socher, R. (2017) · doi:10.18653/v1/w17-4303 · Cited by 13 · see citing works
Supervised Domain Enablement Attention for Personalized Domain Classification
Kim, J.-K., & Kim, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1106 · Cited by 9 · see citing works
Simple Attention-Based Representation Learning for Ranking Short Social Media Posts
Shi, P., Rao, J., & Lin, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1229 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595 · see citing works
Transformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,412 · see citing works
Sanger: A Co-Design Framework for Enabling Sparse Attention using Reconfigurable Architecture
Lu, L., Jin, Y., Bi, H., Luo, Z., Li, P., Wang, T., & Liang, Y. E. (2021) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3466752.3480125 · Cited by 232 · see citing works
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154 · see citing works
ViA: A Novel Vision-Transformer Accelerator Based on FPGA
Wang, T., Gong, L., Wang, C., Yang, Y., Gao, Y., Zhou, X., & Chen, H. (2022) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2022.3197489 · Cited by 150 · see citing works
HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformers
Dong, P., Sun, M., Lu, A., Xie, Y., Liu, K., Kong, Z., Meng, X., Li, Z., Lin, X., Fang, Z., et al. (2023) · 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071047 · Cited by 128 · see citing works
Accelerating Transformer-based Deep Learning Models on FPGAs using Column Balanced Block Pruning
Peng, H., Huang, S., Geng, T., Li, A., Jiang, W., Liu, H., Wang, S., & Ding, C. (2021) · 2021 22nd International Symposium on Quality Electronic Design (ISQED) · doi:10.1109/isqed51717.2021.9424344 · Cited by 105 · see citing works
Self-citationAn Algorithm–Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers
Fang, C., Zhou, A., & Wang, Z. (2022) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2022.3197282 · Cited by 94 · see citing works
Li, Z., Sun, M., Lu, A., Ma, H., Yuan, G., Xie, Y., Tang, H., Li, Y., Leeser, M., Wang, Z., et al. (2022) · 2022 32nd International Conference on Field-Programmable Logic and Applications (FPL) · doi:10.1109/fpl57034.2022.00027 · Cited by 85 · see citing works
Understanding the Potential of FPGA-based Spatial Acceleration for Large Language Model Inference
Chen, H., Zhang, J., Du, Y., Xiang, S., Yue, Z., Zhang, N., Cai, Y., & Zhang, Z. (2024) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3656177 · Cited by 86 · see citing works
Huang, M., Luo, J., Ding, C., Wei, Z., Huang, S., & Yu, H. (2023) · IEEE Transactions on Circuits and Systems I: Regular Papers · doi:10.1109/tcsi.2023.3312775 · Cited by 77 · see citing works
Self-citationFPGA-based Deep Learning Inference Accelerators: Where Are We Standing?
Nechi, A., Groth, L., Mulhem, S., Merchant, F., Buchty, R., & Bereković, M. (2023) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3613963 · Cited by 79 · see citing works
ViTA: A Vision Transformer Inference Accelerator for Edge Applications
Nag, S., Datta, G., Kundu, S., Chandrachoodan, N., & Beerel, P. (2023) · International Symposium on Circuits and Systems · doi:10.1109/iscas46773.2023.10181988 · Cited by 65 · see citing works
Adaptable butterfly accelerator for attention-based NNs via hardware and algorithm co-design
Fan, H., Chau, T., Venieris, S. I., Lee, R., Kouris, A., Luk, W., Lane, N. D., & Abdelfattah, M. S. (2022) · 2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro56248.2022.00050 · Cited by 66 · see citing works
EdgeLLM: A Highly Efficient CPU-FPGA Heterogeneous Edge Accelerator for Large Language Models
Huang, M., Shen, A., Li, K., Peng, H., Li, B., Su, Y., & Yu, H. (2025) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2025.3546256 · Cited by 74 · see citing works
Self-citationAccelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimization
Qi, P., Sha, E. H. ‐., Zhuge, Q., Peng, H., Huang, S., Kong, Z., Song, Y., & Li, B. (2021) · International Conference on Computer Aided Design · doi:10.1109/iccad51958.2021.9643586 · Cited by 55 · see citing works
Self-citationAccelTran: A Sparsity-Aware Accelerator for Dynamic Inference With Transformers
Tuli, S., & Jha, N. K. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3273992 · Cited by 56 · see citing works
Algorithm-hardware Co-design of Attention Mechanism on FPGA Devices
Zhang, X., Wu, Y., Zhou, P., Tang, X., & Hu, J. (2021) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3477002 · Cited by 52 · see citing works
Sarkar, R., Liang, H., Fan, Z., Wang, Z., & Hao, C. (2023) · 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD) · doi:10.1109/iccad57390.2023.10323651 · Cited by 56 · see citing works
A Survey on Hardware Accelerators for Large Language Models
Kachris, C. (2025) · Applied Sciences · doi:10.3390/app15020586 · Cited by 63 · see citing works
Zhao, Z.-Y., Cao, R., Un, K.-F., Yu, W.-H., Mak, P.-I., & Martins, R. (2023) · IEEE Transactions on Circuits and Systems - II - Express Briefs · doi:10.1109/tcsii.2022.3196055 · Cited by 51 · see citing works
Review of neural network model acceleration techniques based on FPGA platforms
Liu, F., Li, H., Hu, W., & He, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128511 · Cited by 41 · see citing works
SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers
Marchisio, A., Dura, D., Capra, M., Martina, M., Masera, G., & Shafique, M. (2023) · 2023 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191521 · Cited by 39 · see citing works
Qi, P., Song, Y., Peng, H., Huang, S., Zhuge, Q., & Sha, E. H.-M. (2021) · Proceedings of the 2021 Great Lakes Symposium on VLSI · doi:10.1145/3453688.3461739 · Cited by 37 · see citing works
MECLA: Memory-Compute-Efficient LLM Accelerator with Scaling Sub-matrix Partition
Qin, Y., Wang, Y., Zhao, Z., Yang, X., Zhou, Y., Wei, S., Hu, Y., & Yin, S. (2024) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca59077.2024.00079 · Cited by 34 · see citing works
SOFA: A Compute-Memory Optimized Sparsity Accelerator via Cross-Stage Coordinated Tiling
Wang, H., Fang, J., Tang, X., Yue, Z., Li, J., Qin, Y., Guan, S., Yang, Q., Wang, Y., Li, C., et al. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00093 · Cited by 34 · see citing works
Self-citationMaxEVA: Maximizing the Efficiency of Matrix Multiplication on Versal AI Engine
Taka, E., Arora, A., Wu, K., & Marculescu, D. (2023) · International Conference on Field-Programmable Technology (FPT) · doi:10.1109/icfpt59805.2023.00016 · Cited by 33 · see citing works
A Fast and Flexible FPGA-based Accelerator for Natural Language Processing Neural Networks
Hur, S., Na, S., Kwon, D., Kim, J., Boutros, A., Nurvitadhi, E., & Kim, J. (2022) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3564606 · Cited by 32 · see citing works
Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLM
Yu, Z., Liang, S., Ma, T., Cai, Y., Nan, Z., Huang, D., Song, X., Hao, Y., Zhang, J., Zhi, T., et al. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00108 · Cited by 31 · see citing works
An Efficient Hardware Accelerator for Sparse Transformer Neural Networks
Fang, C., Guo, S., Wu, W., Lin, J., Wang, Z., Hsu, M. K., & Liu, L. (2022) · 2022 IEEE International Symposium on Circuits and Systems (ISCAS) · doi:10.1109/iscas48785.2022.9937659 · Cited by 30 · see citing works
HPTA: A High Performance Transformer Accelerator Based on FPGA
Han, Y., & Liu, Q. (2023) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl60245.2023.00012 · Cited by 29 · see citing works
Fu, X., Ren, Q., Wu, H., Xiang, F., Luo, Q., Yue, J., Chen, Y., & Zhang, F. (2023) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2023.3315060 · Cited by 28 · see citing works
An End-to-end Multi-task Object Detection using Embedded GPU in Autonomous Driving
Zhou, S., Xie, M., Jin, Y., Miao, F., & Ding, C. (2021) · International Symposium on Quality Electronic Design (ISQED) · doi:10.1109/isqed51717.2021.9424308 · Cited by 28 · see citing works
Self-citationSWAT: An Efficient Swin Transformer Accelerator Based on FPGA
Dong, Q.-W., Xie, X., & Wang, Z.-F. (2024) · Asia and South Pacific Design Automation Conference · doi:10.1109/asp-dac58780.2024.10473931 · Cited by 20 · see citing works
Row-wise Accelerator for Vision Transformer
Wang, H.-Y., & Chang, T.-S. (2022) · International Conference on Artificial Intelligence Circuits and Systems · doi:10.1109/aicas54282.2022.9869928 · Cited by 23 · see citing works
Hardware–Software Co-Design Enabling Static and Dynamic Sparse Attention Mechanisms
Zhao, J., Zeng, P., Shen, G., Chen, Q., & Guo, M. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3373592 · Cited by 25 · see citing works
Exploring HW/SW Co-Design for Video Analysis on CPU-FPGA Heterogeneous Systems
Zhang, X., Ma, Y., Xiong, J., Hwu, W., Kindratenko, V., & Chen, D. (2021) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2021.3093398 · Cited by 24 · see citing works
Jing, L., Dong, C., He, C., Shi, W., & Yin, H. (2022) · Remote Sensing · doi:10.3390/rs14235959 · Cited by 24 · see citing works
ASADI: Accelerating Sparse Attention Using Diagonal-based In-Situ Computing
Li, H., Li, Z., Bai, Z., & Mitra, T. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00065 · Cited by 23 · see citing works
Unified Accelerator for Attention and Convolution in Inference Based on FPGA
Li, T., Zhang, F., Fan, X., Shen, J., Guo, W., & Cao, W. (2023) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas46773.2023.10182145 · Cited by 22 · see citing works
SoftAct: A High-Precision Softmax Architecture for Transformers Supporting Nonlinear Functions
Fu, Y., Zhou, C., Huang, T., Han, E., He, Y., & Jiao, H. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3386779 · Cited by 21 · see citing works
CPSAA: Accelerating Sparse Attention Using Crossbar-Based Processing-In-Memory Architecture
Li, H., Jin, H., Zheng, L., Liao, X., Huang, Y., Liu, C., Xu, J., Duan, Z., Chen, D., & Gui, C. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3344524 · Cited by 20 · see citing works
PICACHU: Plug-In CGRA Handling Upcoming Nonlinear Operations in LLMs
Qin, J., Xia, T., Tan, C., Zhang, J. J., & Zhang, S. Q. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716013 · Cited by 18 · see citing works
EFA-Trans: An Efficient and Flexible Acceleration Architecture for Transformers
Yang, X., & Su, T. (2022) · Electronics · doi:10.3390/electronics11213550 · Cited by 18 · see citing works
A Survey on Sparsity Exploration in Transformer-Based Accelerators
Fuad, K. A. A., & Chen, L. (2023) · Electronics · doi:10.3390/electronics12102299 · Cited by 18 · see citing works
FET-OPU: A Flexible and Efficient FPGA-Based Overlay Processor for Transformer Networks
Bai, Y., Zhou, H., Zhao, K., Wang, H., Chen, J., Yu, J. W., & Wang, K. (2023) · doi:10.1109/iccad57390.2023.10323752 · Cited by 18 · see citing works
Self-citationFan, Z., Zhang, Q., Abillama, P., Shoouri, S., Lee, C. S. G., Blaauw, D. T., Kim, H.-S., & Sylvester, D. (2023) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3579371.3589040 · Cited by 16 · see citing works
Accelerating Transformer Neural Networks on FPGAs for High Energy Physics Experiments
Wojcicki, F., Que, Z., Tapper, A., & Luk, W. W. (2022) · International Conference on Field-Programmable Technology (FPT) · doi:10.1109/icfpt56656.2022.9974463 · Cited by 15 · see citing works
Wang, Z., Wang, G., Jiang, H., Xu, N., & He, G. (2023) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac56929.2023.10247678 · Cited by 15 · see citing works
Qin, Y., Lou, W., Wang, C., Gong, L., & Zhou, X. (2024) · ACM Great Lakes Symposium on VLSI (GLSVLSI) · doi:10.1145/3649476.3658810 · Cited by 15 · see citing works
Efficient memristor accelerator for transformer self-attention functionality
Bettayeb, M., Halawani, Y., Khan, M. U., Saleh, H. H., & Mohammad, B. S. (2024) · Scientific Reports · doi:10.1038/s41598-024-75021-z · Cited by 15 · see citing works
TRAC: Compilation-Based Design of Transformer Accelerators for FPGAs
Plagwitz, P., Hannig, F., & Teich, J. (2022) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl57034.2022.00015 · Cited by 14 · see citing works
LightMamba: Efficient Mamba Acceleration on FPGA with Quantization and Hardware Co-design
Wei, R., Xu, S., Zhong, L., Yang, Z., Guo, Q., Wang, Y., Wang, R., & Li, M. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10993079 · Cited by 14 · see citing works
Accelerating Transformer Networks through Recomposing Softmax Layers
Choi, J., Li, H., Kim, B., Hwang, S., & Ahn, J. H. (2022) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc55918.2022.00018 · Cited by 13 · see citing works
Mobile Transformer Accelerator Exploiting Various Line Sparsity and Tile-Based Dynamic Quantization
Kwon, E., Yoon, J., & Kang, S. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3347291 · Cited by 13 · see citing works
A Cost-Efficient FPGA-Based CNN-Transformer Using Neural ODE
Okubo, I., Sugiura, K., & Matsutani, H. (2024) · IEEE Access · doi:10.1109/access.2024.3480977 · Cited by 12 · see citing works
Sun, X., Zhang, Y., Wang, Q., Zou, X., Liu, Y., Zeng, Z., & Zhuang, H. (2025) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2025.105042 · Cited by 12 · see citing works
LTrans-OPU: A Low-Latency FPGA-Based Overlay Processor for Transformer Networks
Bai, Y., Zhou, H., Zhao, K., Zhang, M., Chen, J., Yu, J., & Wang, K. (2023) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl60245.2023.00048 · Cited by 12 · see citing works
Self-citationWang, H., Wang, Z., Yue, Z., Long, Y., Wei, T., Yang, J., Wang, Y., Li, C., Wei, S., Hu, Y., et al. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756037 · Cited by 11 · see citing works
Energy Efficient FPGA-Based Binary Transformer Accelerator for Edge Devices
Du, C., Ko, S., & Zhang, H. (2024) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas58744.2024.10558631 · Cited by 11 · see citing works
Binary Complex Neural Network Acceleration on FPGA : (Invited Paper)
Peng, H., Zhou, S., Weitze, S., Li, J., Islam, S., Geng, T., Li, A., Zhang, W., Song, M., Xie, M., et al. (2021) · IEEE International Conference on Application-specific Systems, Architectures and Processors (ASAP) · doi:10.1109/asap52443.2021.00021 · Cited by 14 · see citing works
Self-citationGRAEBO: FPGA General Routing Architecture Exploration via Bayesian Optimization
Zheng, S., Qian, J., Zhou, H., & Wang, L. (2022) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl57034.2022.00050 · Cited by 11 · see citing works
Hardware-friendly compression and hardware acceleration for transformer: A survey
Huang, S., Tang, E., Li, S., Ping, X., & Chen, R. (2022) · Electronic Research Archive · doi:10.3934/era.2022192 · Cited by 10 · see citing works
Inference Optimization of Foundation Models on AI Accelerators
Park, Y., Budhathoki, K., Chen, L., Kübler, J. M., Huang, J., Kleindeßner, M., Huan, J., Cevher, V., Wang, Y., & Karypis, G. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671465 · Cited by 11 · see citing works
In-Storage Acceleration of Retrieval Augmented Generation as a Service
Mahapatra, R., Santhanam, H., Priebe, C., Xu, H., & Esmaeilzadeh, H. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731032 · Cited by 10 · see citing works
Rizk, R., Rizk, D., Rizk, F., Kumar, A., & Bayoumi, M. (2022) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas48785.2022.9937531 · Cited by 10 · see citing works
A Self-Attention Network for Deep JSCCM: The Design and FPGA Implementation
Fujimaki, S., Inoue, Y., Hisano, D., Maruta, K., Nakayama, Y., & Hara–Azumi, Y. (2022) · Globecom · doi:10.1109/globecom48099.2022.10001518 · Cited by 10 · see citing works
Calabash: Accelerating Attention Using a Systolic Array Chain on FPGAs
Luo, Z., Lu, L., Jin, Y., Jia, L., & Liang, Y. E. (2023) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl60245.2023.00041 · Cited by 10 · see citing works
BETA: Binarized Energy-Efficient Transformer Accelerator at the Edge
Ji, Y., Fang, C., & Wang, Z. (2024) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas58744.2024.10558636 · Cited by 10 · see citing works
Kwon, E., Song, H., Park, J., & Kang, S. (2023) · Design, Automation, and Test in Europe · doi:10.23919/date56975.2023.10137099 · Cited by 9 · see citing works
EdgeTran: Device-Aware Co-Search of Transformers for Efficient Inference on Mobile Edge Platforms
Tuli, S., & Jha, N. K. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3328287 · Cited by 15 · see citing works
COSA Plus: Enhanced Co-Operative Systolic Arrays for Attention Mechanism in Transformers
Wang, Z., Wang, G., & He, G. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3434447 · Cited by 9 · see citing works
ChatOPU: An FPGA-based Overlay Processor for Large Language Models with Unstructured Sparsity
Zhao, T., Lu, S., Wu, C.-L., & He, L. (2024) · doi:10.1145/3676536.3676761 · Cited by 9 · see citing works
Reconfigurable Stream Network Architecture
Wang, C. C., Zhang, X., Cong, J., & Hoe, J. C. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731088 · Cited by 9 · see citing works
Liao, X., Zhou, T., Zhang, L., Hu, X. Y., & Peng, Y. (2023) · Electronics · doi:10.3390/electronics12020267 · Cited by 9 · see citing works
TransFRU: Efficient Deployment of Transformers on FPGA with Full Resource Utilization
Wang, H., Bai, Y., Yu, J., & Wang, K. (2024) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac58780.2024.10473976 · Cited by 8 · see citing works
Zhang, C., You, Y., Wang, N., Park, J., & Zhang, L. (2025) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2025.3575272 · Cited by 11 · see citing works
Hardware Acceleration of Transformer Networks using FPGAs
Τζάνος, Γ., Kachris, C., & Soudris, D. (2022) · Panhellenic Conference on Electronics & Telecommunications (PACET) · doi:10.1109/pacet56979.2022.9976354 · Cited by 8 · see citing works
Wu, J.-J., Song, M., Zhao, J., Gao, Y.-Z., Li, J., & So, H. K.-H. (2024) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3714416 · Cited by 12 · see citing works
Self-citationHardware Accelerator Design for Sparse DNN Inference and Training: A Tutorial
Mao, W., Wang, M., Xie, X., Wu, X., & Wang, Z. (2023) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2023.3344681 · Cited by 8 · see citing works
ChipMind: LLMs for Agile Chip Design
Firouzi, F., Pan, D. Z., Gu, J., Farahani, B., Chaudhuri, J., Yin, Z., Ma, P., Domanski, P., & Chakrabarty, K. (2025) · IEEE VLSI Test Symposium (VTS) · doi:10.1109/vts65138.2025.11022936 · Cited by 7 · see citing works
Co-Designing Binarized Transformer and Hardware Accelerator for Efficient End-to-End Edge Deployment
Ji, Y., Fang, C., Ma, S., Shao, H., & Wang, Z. (2024) · doi:10.1145/3676536.3676733 · Cited by 7 · see citing works
Byun, W.-H., Woo, J., & Mukhopadhyay, S. (2025) · IEEE Access · doi:10.1109/access.2025.3563196 · Cited by 7 · see citing works
LAD: Efficient Accelerator for Generative Inference of LLM with Locality Aware Decoding
Wang, H., Li, Y., Xu, H., Wang, Y., Liu, L., Yang, J., & Han, Y. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00111 · Cited by 6 · see citing works
Hardware Accelerator for Transformer based End-to-End Automatic Speech Recognition System
Yamini, S., Mirishkar, G. S., Vuppala, A. K., & Purini, S. (2023) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw59300.2023.00027 · Cited by 6 · see citing works
A Hardware Design Framework for Computer Vision Models Based on Reconfigurable Devices
Fan, Z., Hu, W., Liu, F., Xu, D. G., Guo, H., He, Y., & Peng, M. (2023) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3635157 · Cited by 6 · see citing works
Jang, S., & Cho, Y.-B. (2024) · Electronics · doi:10.3390/electronics13030552 · Cited by 6 · see citing works
Energy Efficient FPGA-Based Accelerator for Dynamic Sparse Transformer
Li, Z., Lai, Y., & Zhang, H. (2024) · International Conference on Communications, Circuits and Systems · doi:10.1109/icccas62034.2024.10652850 · Cited by 6 · see citing works
Energy efficient spike transformer accelerator at the edge
Du, C., Wen, Q., Wei, Z., & Zhang, H. (2024) · Intelligent Marine Technology and Systems · doi:10.1007/s44295-024-00040-5 · Cited by 6 · see citing works
Neural architecture search for adversarial robustness via learnable pruning
Li, Y., Zhao, P., Ding, R., Zhou, T. H., Fei, Y., Xu, X., & Lin, X. (2024) · Frontiers in High Performance Computing · doi:10.3389/fhpcp.2024.1301384 · Cited by 6 · see citing works
ProTEA: Programmable Transformer Encoder Acceleration on FPGA
Kabir, E., Bakos, J. D., Andrews, D. A., & Huang, M. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/scw63240.2024.00074 · Cited by 6 · see citing works
Dahal, S., Dhingra, P., Thapa, K. K., Pande, P. P., & Kalyanaraman, A. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3522781 · Cited by 5 · see citing works
Fan, Z., Huang, M., Zhang, X., Liu, R., Lyu, X., Duan, T., Bu, Z., & Liang, J. (2023) · Applied Sciences · doi:10.3390/app132212184 · Cited by 5 · see citing works
Song, H., Xu, L., Wang, Y., Wu, X., Wang, M., & Wang, Z. (2024) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas58744.2024.10557837 · Cited by 5 · see citing works
An FPGA-based Multi-Core Overlay Processor for Transformer-based Models
Lu, S., Zhao, T., Zhang, R., Lin, T.-J., Wu, C.-L., & He, L. (2024) · International Symposium of Electronics Design Automation (ISEDA) · doi:10.1109/iseda62518.2024.10617729 · Cited by 5 · see citing works
Wang, X., Miao, S., Qu, P., & Zhang, Y. (2024) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl64840.2024.00054 · Cited by 5 · see citing works
Low latency transformer inference on FPGAs for physics applications with hls4ml
Jiang, Z., Yin, D., Chen, Y.-H., Khoda, E. E., Hauck, S., Hsu, S.-C., Govorkova, E., Harris, P. C., Lončar, V., & Moreno, E. A. (2025) · Journal of Instrumentation · doi:10.1088/1748-0221/20/04/p04014 · Cited by 5 · see citing works
Enabling Energy-Efficient Inference for Self-Attention Mechanisms in Neural Networks
Chen, Q., Sun, C., Lu, Z., & Gao, C. (2022) · IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) · doi:10.1109/aicas54282.2022.9869924 · Cited by 5 · see citing works
Real block-circulant matrices and DCT-DST algorithm for transformer neural network
Asriani, E., Muchtadi-Alamsyah, I., & Purwarianti, A. (2023) · Frontiers in Applied Mathematics and Statistics · doi:10.3389/fams.2023.1260187 · Cited by 5 · see citing works
An FPGA-Based Efficient Streaming Vector Processing Engine for Transformer-Based Models
He, Z., Zhao, T., Miao, S., Wu, C., & He, L. (2024) · International Symposium of Electronics Design Automation (ISEDA) · doi:10.1109/iseda62518.2024.10617499 · Cited by 5 · see citing works
- Authors
- Li, B., Pandey, S., Fang, H., Lyv, Y., Li, J., Chen, J., Xie, M., Wan, L., Liu, H., & Ding, C.
- Affiliations
- University of Connecticut (US)
- Stevens Institute of Technology (US)
- Syracuse University (US)
- Microsoft (United States) (US)
- Oak Ridge National Laboratory (US)
- The University of Texas at San Antonio (US)
- ORCID
- Venue
- Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design
- Publisher
- ACM
- Published
- 2020-08-10
- Pages
- 175-180
- OpenAlex
- W3047848469
- Cited by
- 206
- References
- 14
- Citations by year
- 2026: 23 · 2025: 54 · 2024: 55 · 2023: 38 · 2022: 25 · 2021: 11
- FWCI
- 13.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium on Low Power Electronics and Design (ISLPED) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Electrical and Electronic Engineering › VLSI and FPGA Design Techniques
- Keywords
- transformer models, FPGA acceleration, model compression, block circulant matrices, self-attention, energy efficiency, hardware architecture design, deep learning acceleration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
