Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers
Open accessAbstract
Transformers have transformed the field of natural language processing. Their superior performance is largely attributed to the use of stacked “self-attention” layers, each of which consists of matrix multiplies as well as softmax operations. As a result, unlike other neural networks, the softmax operation accounts for a significant fraction of the total run-time of Transformers. To address this, we propose Softermax, a hardware-friendly softmax design. Softermax consists of base replacement, low-precision softmax computations, and an online normalization calculation. We show Softermax results in 2.35x the energy efficiency at 0.90x the size of a comparable baseline, with negligible impact on network accuracy.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Non-linear operations like Softmax, LayerNorm, and GELU present computational challenges on conventional hardware due to their complexity, potentially restricting end-to-end transformer inference.
“Non-linear operations like Softmax, LayerNorm, and GELU, though less prevalent, present computational challenges on conventional hardware due to their complexity, potentially restricting the enhancement of end-to-end transformer inference [31].”
From Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey (2024) · cites this paper as [31]
The “Softmax Bottlenecks” section (page not supplied) explains why softmax is inefficient on CPUs and GPUs, and the Conclusion says its time and energy costs limit Transformer implementations. The paper does not establish the same hardware challenges for LayerNorm or GELU, so the claim’s broader list of operations is not fully supported. (LLM, conf=0.88)
Experimental — accuracy not yet measuredA large portion of the inference time in ViT models is consumed by floating-point non-linear activations, normalizations, and the associated quantization and dequantization operations.
“A large portion of the inference time in ViT models is consumed by floating-point non-linear activations, normalizations, and the associated quantization and dequantization operations [31].”
From Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey (2024) · cites this paper as [31]
The Introduction says attention operations, particularly softmax, account for a large fraction of Transformer runtime; its runtime example is BERT-Large, not a ViT model. The Softmax Bottlenecks section also notes a possible slight performance penalty from casting between data types. The paper does not establish that floating-point activations, normalizations, and quantization/dequantization operations collectively consume a large portion of ViT inference time. (LLM, conf=0.86)
Experimental — accuracy not yet measuredVarious approximation methods that hold floating-point arithmetic lower certain computational costs but cannot meet the demands of integer arithmetic.
“In addition, various approximation methods that hold floating-point arithmetic are presented [36, 50]; while they lower certain computational costs, they cannot meet the demands of integer arithmetic.”
From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [36]
In “Low-precision Softmax Operations” (page unavailable), the paper says 32-bit floating-point units are costly in area and power and proposes lower-precision operations. “Softmax Bottlenecks” discusses earlier approximate units, but does not establish that those methods retain floating-point arithmetic or cannot meet integer-arithmetic requirements. The Experimental Setup instead describes Softermax’s compatibility with 8-bit integer datapaths. (LLM, conf=0.78)
Experimental — accuracy not yet measuredSoftmax cannot follow the dyadic arithmetic pipeline due to non-linearity, and the exponential arithmetic in Softmax is typically unsupported by integer-only logic units.
“Due to the non-linearity, Softmax cannot follow the dyadic arithmetic pipeline discussed above, and the exponential arithmetic in Eq. 11 is typically unsupported by integer-only logic units [36].”
From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [36]
The Softermax Software section describes the natural exponential as nonlinear and says it creates hardware overhead. Low-precision Softmax Operations says commodity hardware does not support low-precision special-function units such as power and division. The paper does not specifically say that Softmax cannot follow a dyadic arithmetic pipeline or that exponential arithmetic is unsupported by integer-only logic units (page numbers unavailable in our copy). (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537 · see citing works
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · see citing works
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N. P., Patterson, D. A., Agrawal, G., Bajwa, R. S., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · ACM SIGARCH Computer Architecture News · doi:10.1145/3140659.3080246 · Cited by 1,431 · see citing works
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · see citing works
Zafrir, O., Boudoukh, G., Izsak, P., & Wasserblat, M. (2019) · 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition (EMC2-NIPS) · doi:10.1109/emc2-nips53020.2019.00016 · Cited by 442 · see citing works
Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
Wu, H., Judd, P., Zhang, X., Isaev, M., & Micikevicius, P. (2020) · arXiv · doi:10.48550/arxiv.2004.09602 · Cited by 246 · see citing works
 3: Accelerating attention mechanisms in neural networks with approximation
Ham, T. J., Jung, S. J., Kim, S., Oh, Y. H., Park, Y., Song, Y., Park, J.-H., Lee, S., Park, K., Lee, J. W., et al. (2020) · 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca47549.2020.00035 · Cited by 242 · see citing works
Guo, Q., Qiu, X., Liu, P., Shao, Y., Xue, X., & Zhang, Z. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1133 · Cited by 158 · see citing works
MAGNet: A Modular Accelerator Generator for Neural Networks
Venkatesan, R., Shao, Y. S., Wang, M., Clemons, J., Dai, S. X., Fojtik, M. R., Keller, B., Klinefelter, A., Pinckney, N., Raina, P., et al. (2019) · doi:10.1109/iccad45719.2019.8942127 · Cited by 142 · see citing works
Self-citationEfficient Precision-Adjustable Architecture for Softmax Function in Deep Learning
Zhu, D., Lu, S., Wang, M., Lin, J., & Wang, Z. (2020) · IEEE Transactions on Circuits and Systems II: Express Briefs · doi:10.1109/tcsii.2020.3002564 · Cited by 115 · see citing works
Design and Implementation of an Approximate Softmax Layer for Deep Neural Networks
Gao, Y., Liu, W., & Lombardi, F. (2020) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas45731.2020.9180870 · Cited by 82 · see citing works
Fully Quantized Transformer for Machine Translation
Prato, G., Charlaix, E., & Rezagholizadeh, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.1 · Cited by 67 · see citing works
Efficient Softmax Hardware Architecture for Deep Neural Networks
Du, G., Tian, C., Li, Z., Zhang, D., Yin, Y., & Ouyang, Y. (2019) · ACM Great Lakes Symposium on VLSI (GLSVLSI) · doi:10.1145/3299874.3317988 · Cited by 53 · see citing works
Towards Fully 8-bit Integer Inference for the Transformer Model
Lin, Y., Li, Y., Liu, T., Xiao, T., Liu, T., & Zhu, J. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/520 · Cited by 30 · see citing works
Online normalizer calculation for softmax
Milakov, M., & Gimelshein, N. (2018) · arXiv · doi:10.48550/arxiv.1805.02867 · Cited by 27 · cited 2× · see citing works
“Prior work [18] has proposed an “online normalizer” calculation, where the max is calculated continuously along with the normalization value ( i.e , the summation for the denominator).”
BackgroundOptimizing Transformers with Approximate Computing for Faster, Smaller and more Accurate NLP ModelsPreprint
Nagarajan, A., Sen, S., Stevens, J. R., & Raghunathan, A. (2021) · arXiv (Cornell University) · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
2 of the 23 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- A. Radford, J. Wu, R. Child et al., "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019.
Cited by
Sorted by citations
Yenduri, G., Ramalingam, M., Selvi, G. C., Supriya, Y., Srivastava, G., Maddikunta, P. K. R., Raj, G. D., Jhaveri, R. H., Prabadevi, B., Wang, W., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3389497 · Cited by 759 · see citing works
I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference
Li, Z., & Gu, Q. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01565 · Cited by 141 · see citing works
“11 is typically unsupported by integer-only logic units [36].”
BackgroundMethodX-Former: In-Memory Acceleration of Transformers
Sridharan, S., Stevens, J. R., Roy, K., & Raghunathan, A. (2023) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2023.3282046 · Cited by 70 · see citing works
Self-citationA 95.6-TOPS/W Deep Learning Inference Accelerator With Per-Vector Scaled 4-bit Quantization in 5 nm
Keller, B., Venkatesan, R., Dai, S. X., Tell, S. G., Zimmer, B., Sakr, C., Dally, W. J., Gray, C. T., & Khailany, B. (2023) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2023.3234893 · Cited by 61 · see citing works
Self-citationSaha, S., & Xu, L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130417 · Cited by 66 · see citing works
Compute-in-Memory Technologies and Architectures for Deep Learning Workloads
Ali, M., Roy, S., Saxena, U., Sharma, T., Raghunathan, A., & Roy, K. (2022) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2022.3203583 · Cited by 54 · see citing works
Self-citationSOLE: Hardware-Software Co-design of Softmax and LayerNorm for Efficient Transformer Inference
Wang, W., Zhou, S., Sun, W., Sun, P., & Liu, Y. (2023) · 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD) · doi:10.1109/iccad57390.2023.10323725 · Cited by 51 · see citing works
Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputation
Yazdanbakhsh, A., Moradifirouzabadi, A., Li, Z., & Kang, M.-G. (2022) · Micro · doi:10.1109/micro56248.2022.00059 · Cited by 53 · see citing works
An Empirical Evaluation of Enhanced Performance Softmax Function in Deep Learning
Mehra, S., Raut, G., Purkayastha, R. D., Vishvakarma, S. K., & Biasizzo, A. (2023) · IEEE Access · doi:10.1109/access.2023.3265327 · Cited by 34 · see citing works
MEGA: A Memory-Efficient GNN Accelerator Exploiting Degree-Aware Mixed-Precision Quantization
Zhu, Z., Li, F., Li, G., Liu, Z., Mo, Z., Hu, Q., Liang, X., & Cheng, J. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00020 · Cited by 30 · see citing works
SimA: Simple Softmax-free Attention for Vision Transformers
Koohpayegani, S. A., & Pirsiavash, H. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00259 · Cited by 40 · see citing works
Flex-PE: Flexible and SIMD Multiprecision Processing Element for AI Workloads
Lokhande, M., Raut, G., & Vishvakarma, S. K. (2025) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2025.3553069 · Cited by 29 · see citing works
Hardware-efficient Softmax Approximation for Self-Attention Networks
Koca, N. A., Do, A. T., & Chang, C.-H. (2023) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas46773.2023.10181465 · Cited by 29 · see citing works
Fu, X., Ren, Q., Wu, H., Xiang, F., Luo, Q., Yue, J., Chen, Y., & Zhang, F. (2023) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2023.3315060 · Cited by 28 · see citing works
From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks
Geng, X., Wang, Z., Chen, C., Xu, Q., Xu, K., Jin, C., Gupta, M., Yang, X., Chen, Z., Sabry Aly, M. M., et al. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3394494 · Cited by 26 · see citing works
ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers
İslamoğlu, G., Scherer, M., Paulin, G., Fischer, T., Jung, V. J. B., Garofalo, A., & Luca Benini, C. (2023) · International Symposium on Low Power Electronics and Design (ISLPED) · doi:10.1109/islped58423.2023.10244348 · Cited by 26 · see citing works
Hardware–Software Co-Design Enabling Static and Dynamic Sparse Attention Mechanisms
Zhao, J., Zeng, P., Shen, G., Chen, Q., & Guo, M. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3373592 · Cited by 25 · see citing works
SoftAct: A High-Precision Softmax Architecture for Transformers Supporting Nonlinear Functions
Fu, Y., Zhou, C., Huang, T., Han, E., He, Y., & Jiao, H. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3386779 · Cited by 21 · see citing works
TEA-S: A Tiny and Efficient Architecture for PLAC-Based Softmax in Transformers
Mei, Z., Dong, H., Wang, Y., & Pan, H. (2023) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2023.3265710 · Cited by 20 · see citing works
MINOTAUR: A Posit-Based 0.42–0.50-TOPS/W Edge Transformer Inference and Training Accelerator
Prabhu, K., Radway, R. M., Yu, J. J., Bartolone, K., Giordano, M., Peddinghaus, F., Urman, Y., Khwa, W.-S., Chih, Y.-D., Chang, M., et al. (2025) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2025.3545731 · Cited by 19 · see citing works
Memristive dynamics enabled neuromorphic computing systems
Yan, B., Yang, Y., & Huang, R. (2023) · Science China Information Sciences · doi:10.1007/s11432-023-3739-0 · Cited by 19 · see citing works
PICACHU: Plug-In CGRA Handling Upcoming Nonlinear Operations in LLMs
Qin, J., Xia, T., Tan, C., Zhang, J. J., & Zhang, S. Q. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716013 · Cited by 18 · see citing works
AI Computing Systems for Large Language Models Training
Zhang, Z., Wen, Y., Lyu, H.-Q., Liu, C., Zhang, R., Li, X., Wang, C., Du, Z., Guo, Q., Li, L., et al. (2025) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-4178-1 · Cited by 19 · see citing works
Mu, C., Huang, Z., Jiang, H., Liao, J., Zhou, Y. L., Chen, L., Zhang, Y., Zhu, H., Yang, J., Liu, Q., et al. (2025) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2025.3577335 · Cited by 17 · see citing works
Reusing Softmax Hardware Unit for GELU Computation in Transformers
Peltekis, C., Alexandridis, K., & Dimitrakopoulos, G. (2024) · IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) · doi:10.1109/aicas59952.2024.10595882 · Cited by 16 · see citing works
ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameters
Liu, S., Tao, G., Zou, Y., Chow, D., Fan, Z., Lei, K., Pan, B., Sylvester, D., Kielian, G., & Saligane, M. (2024) · doi:10.1145/3676536.3676766 · Cited by 15 · see citing works
Wang, J., Zhang, L., Li, X., Yang, H., & Liu, Y. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3329039 · Cited by 15 · see citing works
Zhan, Y., Yu, W.-H., Un, K.-F., Martins, R. P., & Mak, P. (2024) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2024.3497187 · Cited by 14 · see citing works
Accelerating Transformer Networks through Recomposing Softmax Layers
Choi, J., Li, H., Kim, B., Hwang, S., & Ahn, J. H. (2022) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc55918.2022.00018 · Cited by 13 · see citing works
TCB: Accelerating Transformer Inference Services with Request Concatenation
Fu, B., Chen, F., Li, P., & Zeng, D. (2022) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3545008.3545052 · Cited by 13 · see citing works
Softmax Bias Correction for Quantized Generative Models
Pandey, N. P., Fournarakis, M., Patel, C., & Nagel, M. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00157 · Cited by 13 · see citing works
High-throughput systolic array-based accelerator for hybrid transformer-CNN networks
Song, Q., Dai, Y., Lu, H., & Jin, G. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.102194 · Cited by 13 · see citing works
OPAL: Outlier-Preserved Microscaling Quantization Accelerator for Generative Large Language Models
Koo, J., Park, D., Jung, S., & Kung, J. (2024) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3649329.3657323 · Cited by 12 · see citing works
Hardware-oriented algorithms for softmax and layer normalization of large language models
Li, W., Lyu, D., Wang, G., Hu, A., Xu, N., & He, G. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4137-4 · Cited by 12 · see citing works
Hardware Accelerator for Approximation-Based Softmax and Layer Normalization in Transformers
Kim, R., Lee, D., Kim, J., Park, J., & Lee, S. E. (2025) · Electronics · doi:10.3390/electronics14122337 · Cited by 10 · see citing works
Range-Invariant Approximation of Non-Linear Operations for Efficient BERT Fine-Tuning
Kim, J., Lee, J., Choi, J., Han, J., & Lee, S. (2023) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac56929.2023.10247958 · Cited by 9 · see citing works
Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoT
Ling, T., Qian, C., & Schiele, G. (2024) · IEEE Annual Congress on Artificial Intelligence of Things (AIoT) · doi:10.1109/aiot63253.2024.00017 · Cited by 9 · see citing works
Topkima-Former: Low-Energy, Low-Latency Inference for Transformers Using Top- k In-Memory ADC
Dong, S., Yang, J., Peng, X., Shang, H., Ke, Y., Yang, X., Liu, H., & Basu, A. (2025) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2025.3549060 · Cited by 10 · see citing works
Wang, J., Lou, M., Yanq, Z., Peng, R., Li, H., Dong, W.-K., Lyu, H., Li, Y., Li, J., Yu, H., et al. (2025) · Custom Integrated Circuits Conference · doi:10.1109/cicc63670.2025.10983287 · Cited by 9 · see citing works
An 8-bit Single Perceptron Processing Unit for Tiny Machine Learning Applications
Crepaldi, M., Salvo, M. D., & Merello, A. (2023) · IEEE Access · doi:10.1109/access.2023.3327517 · Cited by 9 · see citing works
STAR: An Efficient Softmax Engine for Attention Model with RRAM Crossbar
Zhai, Y., Li, B., Yan, B., & Wang, J. (2023) · Design, Automation, and Test in Europe · doi:10.23919/date56975.2023.10137271 · Cited by 8 · see citing works
IRET: Incremental Resolution Enhancing Transformer
Latibari, B. S., Salehi, S., Homayoun, H., & Sasan, A. (2024) · ACM Great Lakes Symposium on VLSI (GLSVLSI) · doi:10.1145/3649476.3660380 · Cited by 8 · see citing works
Wu, J.-J., Song, M., Zhao, J., Gao, Y.-Z., Li, J., & So, H. K.-H. (2024) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3714416 · Cited by 12 · see citing works
Optimizing Vision Transformers: Unveiling ‘Focus and Forget’ for Enhanced Computational Efficiency
Latibari, B. S., Homayoun, H., & Sasan, A. (2025) · IEEE Access · doi:10.1109/access.2025.3540399 · Cited by 9 · see citing works
Wang, W., Sun, W., & Liu, Y. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3488572 · Cited by 7 · see citing works
Algorithm-Hardware Co-Design of a Unified Accelerator for Non-Linear Functions in Transformers
Du, H., Wen, C., Chen, Z., Zhang, L., Sun, Q., Yan, Z., & Zhuo, C. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10992717 · Cited by 7 · see citing works
Efficient Softmax Approximation for Deep Neural Networks with Attention MechanismPreprint
Vasyltsov, I., & Chang, W. (2021) · arXiv.org · doi:10.48550/arxiv.2111.10770 · Cited by 25 · see citing works
MBS: A High-Precision Approximation Method for Softmax and Efficient Hardware Implementation
Wu, Y., Xie, Z., Pan, H., & Wang, Y. (2025) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2025.3559069 · Cited by 6 · see citing works
Kim, D. E., Sharma, T., & Roy, K. (2025) · IEEE transactions on circuits and systems for artificial intelligence. · doi:10.1109/tcasai.2025.3601975 · Cited by 6 · see citing works
GenAI at the Edge: Comprehensive Survey on Empowering Edge DevicesPreprint
Navardi, M., Aalishah, R., Fu, Y., Lin, Y., Li, H., Chen, Y., & Mohsenin, T. (2025) · Qeios · doi:10.32388/jeu3u0 · Cited by 6 · see citing works
VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformers
Wang, R., İslamoğlu, G., Belano, A., Potocnik, V., Conti, F., Garofalo, A., & Luca Benini, C. (2025) · IEEE Symposium on Computer Arithmetic (ARITH) · doi:10.1109/arith64983.2025.00016 · Cited by 5 · see citing works
BBAL: A Bidirectional Block Floating Point-Based Quantisation Accelerator for Large Language Models
Han, X., Cheng, Y., Wang, J., Lu, J., Wang, H., Zhang, X. X., Xu, N., Yang, D., & Jiang, Z. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132978 · Cited by 5 · see citing works
A Case for Low Bitwidth Floating Point Arithmetic on FPGA for Transformer Based DNN Inference
Wu, J., Song, M., Zhao, J., & So, H. K. (2024) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw63119.2024.00045 · Cited by 5 · see citing works
Wang, X., Miao, S., Qu, P., & Zhang, Y. (2024) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl64840.2024.00054 · Cited by 5 · see citing works
Understanding the Performance Horizon of the Latest ML Workloads with NonGEMM Workloads
Karami, R., Kao, S.-C., & Kwon, H. (2025) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass64960.2025.00011 · Cited by 5 · see citing works
Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey
Du, D., Gong, G., & Chu, X. (2024) · arXiv · doi:10.48550/arxiv.2405.00314 · Cited by 4 · see citing works
“Non-linear operations like Softmax, LayerNorm, and GELU, though less prevalent, present computational challenges on conventional hardware due to their complexity, potentially restricting the enhancement of end-to-end trans-former inference [31].”
BackgroundPIVOT- Input-aware Path Selection for Energy-efficient ViT Inference
Moitra, A., Bhattacharjee, A., & Panda, P. (2024) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3649329.3655679 · Cited by 4 · see citing works
Optoelectronic nonlinear Softmax operator based on diffractive neural networks
Zhan, Z., Wang, H., Liu, Q., & Fu, X. (2024) · Optics Express · doi:10.1364/oe.527843 · Cited by 4 · see citing works
HAAN: A Holistic Approach for Accelerating Normalization Operations in Large Language Models
Peng, T., Xia, T., Qin, J., & Zhang, S. Q. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10993005 · Cited by 4 · see citing works
A Unified Accelerator for All-in-One Image Restoration Based on Prompt Degradation Learning
Zhang, S., Dong, Q., Mao, W., & Wang, Z. (2025) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2024.3519532 · Cited by 4 · see citing works
Taghavizade, A., Rahmati, D., Gorgin, S., & Lee, J. (2024) · IEEE International System-on-Chip Conference (SOCC) · doi:10.1109/socc62300.2024.10737795 · Cited by 4 · see citing works
Bit-slice Architecture for DNN Acceleration with Slice-level Sparsity Enhancement and Exploitation
Choi, I., Yoon, Y.-S., & Yang, J.-S. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00067 · Cited by 3 · see citing works
SoftmAP: Software-Hardware Co-Design for Integer-Only Softmax on Associative Processors
Rakka, M., Li, J., Dai, G., Eltawil, A. M., Fouda, M. E., & Kurdahi, F. J. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10992862 · Cited by 3 · see citing works
A Low-Complexity and Reconfigurable Design for Nonlinear Function Approximation in Transformers
Wu, Q.-X., Teng, S.-S., Shieh, M., Huang, C.-T., & Lu, J. (2025) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2025.3585109 · Cited by 3 · see citing works
Hsieh, M.-H., Li, X.-H., Huang, Y.-H. J., Kuo, P.-H., & Huang, J.-D. (2024) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas58744.2024.10558086 · Cited by 3 · see citing works
Attar: RRAM-based in-memory attention accelerator with software-hardware co-optimization
Li, B., Qi, Y., Wang, Y., & Han, Y. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4247-4 · Cited by 3 · see citing works
RISC-V-Based GPGPU With Vector Capabilities for High-Performance Computing
Li, J., Yu, F., Ma, M., Liu, W., Wang, Y., Wu, H., & He, H. (2025) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2025.3574427 · Cited by 3 · see citing works
TReX- Reusing Vision Transformer’s Attention for Efficient Xbar-Based Computing
Moitra, A., Bhattacharjee, A., Kim, Y., & Panda, P. (2024) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2024.3480524 · Cited by 3 · see citing works
Zhou, H., Liu, Y., Wang, H., Tang, E., Li, S., Zhang, Y., Dai, G., Liu, Y., & Wang, K. (2025) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1145/3658617.3697788 · Cited by 3 · see citing works
A Low-complexity and Reconfigurable Design for Nonlinear Function Approximation in Transformers
Wu, Q., Teng, S.-S., Shieh, M., Huang, C.-T., & Lu, J. (2025) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas56072.2025.11044081 · Cited by 3 · see citing works
Shao, H., & Wang, Z. (2025) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2025.3561000 · Cited by 2 · see citing works
An Area-Efficient Accelerator for Convolution and Vision Transformer
Hsieh, P.-C., Xu, Y.-Z., & Fan, Y. (2025) · IEEE International Conference on Consumer Technology (ICCT-Pacific) · doi:10.1109/icct-pacific63901.2025.11012844 · Cited by 2 · see citing works
Efficient FPGA Implementation of Time-Domain Popcount for Low-Complexity Machine Learning
Duan, S., Sartori, M. L. L., Shafik, R., Yakovlev, A., & Özer, E. (2025) · IEEE International Symposium on Asynchronous Circuits and Systems (ASYNC) · doi:10.1109/async65240.2025.00012 · Cited by 2 · see citing works
Wang, Z., Fan, H., Waris, H., Wang, G., Li, Z., Jiang, J., Sun, Y., & He, G. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132861 · Cited by 2 · see citing works
MATA: A Memory-Efficient Attention Accelerator for LLMs Exploiting Look-Back KV Cache Pruning
Zhu, Z., Li, G., Liu, T., Mo, Z., Liang, X., & Cheng, J. (2025) · IEEE Transactions on Computers · doi:10.1109/tc.2025.3626150 · Cited by 2 · see citing works
Ahmed, S., Li, J., Zhuang, W., Chen, C., & Lyu, L. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01968 · Cited by 2 · see citing works
NOVA: NoC-based Vector Unit for Mapping Attention Layers on a CNN Accelerator
Upadhyay, M., Juneja, R., Wong, W., & Peh, L.-S. (2024) · Design, Automation, and Test in Europe · doi:10.23919/date58400.2024.10546727 · Cited by 2 · see citing works
An Energy-Efficient Architecture of Approximate Softmax Functions for Transformer in Edge Computing
Li, S., Yin, B., & Zhang, H. (2023) · International Conference on Electronics and Electrical Engineering Technology (EEET) · doi:10.1109/eeet61723.2023.00015 · Cited by 2 · see citing works
Vision Transformer Inference on a CNN Accelerator
Yi, C., Moh, H., & Ha, S. (2024) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd63220.2024.00101 · Cited by 2 · see citing works
Serving Transformer Models via Joint Requst Scheduling and Batching in the Network Edge
Fu, B., Chen, F., Li, P., & Zeng, D. (2025) · IEEE Transactions on Sustainable Computing · doi:10.1109/tsusc.2025.3528105 · Cited by 2 · see citing works
An Efficient Training Architecture for Nonlinear Softmax Function in Transformers
Shao, H., & Wang, Z. (2025) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas56072.2025.11044130 · Cited by 2 · see citing works
Low-Cost FlashAttention with Fused Exponential and Multiplication Hardware Operators
Alexandridis, K., Titopoulos, V., & Dimitrakopoulos, G. (2025) · IEEE Computer Society Annual Symposium on VLSI · doi:10.1109/isvlsi65124.2025.11130263 · Cited by 6 · see citing works
Colagrande, L., & Luca Benini, C. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132520 · Cited by 2 · see citing works
Hun, R., Xu, J., Sun, M., Zhang, S., Chen, Y., & Chiang, P. Y. (2022) · International Conference on Communications, Circuits and Systems · doi:10.1109/icccas55266.2022.9825471 · Cited by 2 · see citing works
APTO: Accelerating Serialization-Based Point Cloud Transformers with Position-Aware Pruning
Sun, Q., Meng, R., Fan, H., Ding, F., Lu, L., Wu, J., Li, X., & Yan, G. (2025) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1145/3658617.3697655 · Cited by 1 · see citing works
Park, J.-S., Lee, T., Lee, H., Park, C., Cho, Y., Kang, M., Lee, H., Kang, J., Jeon, T., Lee, D., et al. (2025) · Digest of technical papers/Digest of technical papers - IEEE International Solid-State Circuits Conference · doi:10.1109/isscc49661.2025.10904722 · Cited by 1 · see citing works
Asynchronous Design of a Bitwise Elimination Argmax via High-Level Modeling in GraphRack
Squires-Parkin, H., Chan, A., Shafik, R., Wheeldon, A., & Yakovlev, A. (2025) · IEEE International Symposium on Asynchronous Circuits and Systems (ASYNC) · doi:10.1109/async65240.2025.00021 · Cited by 1 · see citing works
Heo, J., Rho, S., Kim, K., & Chung, K.-S. (2025) · IEEE Symposium in Low-Power and High-Speed Chips (COOL CHIPS) · doi:10.1109/coolchips65488.2025.11018597 · Cited by 1 · see citing works
An Integer-only Quantization Framework for Edge Deployment of Large Language Models
Hu, Y., Yuan, Z., Gao, W., Zhang, S., & Liu, Y. (2025) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas56072.2025.11043993 · Cited by 1 · see citing works
TEA-SPS: A Tiny and Efficient Architecture for Softmax With Parallelism and Sparsity Adaptability
Cui, Z., Ma, Y., Cheng, Y., Mei, Z., Yan, F., Pan, H., & Wang, Y. (2025) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2025.3610210 · Cited by 1 · see citing works
Hardware-Oriented and Precisely Approximated Online Softmax for Deep Learning Models
Hirayae, S., Yoshioka, K., Tanaka, Y., & Tamukoh, H. (2025) · IEEE International Conference on Electronics, Circuits and Systems (ICECS) · doi:10.1109/icecs66544.2025.11270517 · Cited by 1 · see citing works
Prasad, A. S., İslamoğlu, G., Bertaccini, L., Rossi, D., Conti, F., & Benini, L. (2025) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd65941.2025.00022 · Cited by 1 · see citing works
Sparse Approximation of Softmax: Hardware-Efficient Acceleration for Long Sequence Inference
Ma, L., Zhao, Z., Wu, X., Chen, G. P., & Yin, W. (2025) · IEEE International Conference on ASIC (ASICON) · doi:10.1109/asicon66040.2025.11326470 · Cited by 1 · see citing works
Wang, J., Lou, M., Yang, Z., Li, Z., Li, J., Li, Y., Yoo, J., & Lin, L. (2026) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2026.3654682 · Cited by 1 · see citing works
Vectorized FlashAttention with low-cost exponential computation in RISC-V vector processors
Titopoulos, V., Alexandridis, K., & Dimitrakopoulos, G. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-026-08322-x · Cited by 1 · see citing works
ReconFormer: A Multi-Level Run-Time Reconfigurable System-on-Chip for Accelerating Transformers
Yang, J., Tombesi, G., Zuckerman, J., & Carloni, L. P. (2025) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl68686.2025.00014 · Cited by 1 · see citing works
Hardware-Efficient Softmax and Layer Normalization with Guaranteed Normalization for Edge Devices
Choi, D., Kim, H., & Kim, J. (2026) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas66217.2026.11562304 · Cited by 1 · see citing works
Activation Function Integration for Accelerating Multi-Layer Graph Convolutional Neural Networks
Grailoo, M., Nikoubin, T., Gustafsson, O., & Nunez-Yanez, J. L. (2024) · IEEE Dallas Circuits and Systems Conference (DCAS) · doi:10.1109/dcas61159.2024.10539892 · Cited by 1 · see citing works
Liu, H., Wang, Y., Huang, Z., Fan, S., Tang, C., Zhang, S., Lei, L., Yang, H., Liu, Y., & Jia, H. (2024) · IEEE International System-on-Chip Conference (SOCC) · doi:10.1109/socc62300.2024.10737793 · Cited by 1 · see citing works
Meng, Z., Shu, L., Zeng, J., Li, Z., Lv, K., Yang, H., & Hao, J. (2024) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2024.3488782 · Cited by 1 · see citing works
- Authors
- Stevens, J. R., Venkatesan, R., Dai, S., Khailany, B., & Raghunathan, A.
- Affiliations
- Purdue University West Lafayette (US)
- Nvidia (United Kingdom) (GB)
- Venue
- 2021 58th ACM/IEEE Design Automation Conference (DAC)
- Publisher
- IEEE
- Published
- 2021-12-05
- Pages
- 469-474
- ISSN
- 0738-100X (ISSN-L)
- OpenAlex
- W3137649401
- Cited by
- 148
- References
- 23
- Citations by year
- 2026: 22 · 2025: 68 · 2024: 35 · 2023: 16 · 2022: 5 · 2021: 2
- FWCI
- 4.74
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- ACM/IEEE Design Automation Conference (DAC) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
- Keywords
- softmax, transformer models, hardware-software co-design, self-attention, low-precision arithmetic, energy efficiency, neural network acceleration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
