Accelerating Sparse Deep Neural Networks
Open accessAbstract
As neural network model sizes have dramatically increased, so has the interest in various techniques to reduce their parameter counts and accelerate their execution. An active area of research in this field is sparsity - encouraging zero values in parameters that can then be discarded from storage or computations. While most research focuses on high levels of sparsity, there are challenges in universally maintaining model accuracy as well as achieving significant speedups over modern matrix-math hardware. To make sparsity adoption practical, the NVIDIA Ampere GPU architecture introduces sparsity support in its matrix-math units, Tensor Cores. We present the design and behavior of Sparse Tensor Cores, which exploit a 2:4 (50%) sparsity pattern that leads to twice the math throughput of dense matrix units. We also describe a simple workflow for training networks that both satisfy 2:4 sparsity pattern requirements and maintain accuracy, verifying it on a wide range of common tasks and model architectures. This workflow makes it easy to prepare accurate models for efficient deployment on Sparse Tensor Cores.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Mishra et al.'s resultant vanilla transformer and BERT models are 8x smaller than the original model, with no loss in accuracy.
“The resultant vanilla transformer and BERT models are 8x smaller than the original model, with no loss in accuracy.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [86]
Section 5.4 (pp. 12–13) reports that the sparse Transformer and BERT models match the dense models’ accuracy. But the Introduction (p. 1) says the paper’s 2:4 sparsity pattern halves the parameter count; it does not support the claim that these models are 8× smaller. (LLM, conf=0.90)
Experimental — accuracy not yet measuredThe tensor core in Nvidia A100 GPU can accelerate the 2:4 sparsity pattern by a factor of 2.
“the tensor core in Nvidia A100 GPU can accelerate the 2:4 sparsity pattern, illustrated in Figure 9(e), by a factor of 2 [86].”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [86]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1× · see citing works
“Table 2 shows the accuracy of a wide variety of networks: popular networks like ResNet [42], VGG [43] and Inception [38], stacked U-Nets (SUNet) [44], dilated residual Table 3.”
BackgroundVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · cited 1× · see citing works
“Table 2 shows the accuracy of a wide variety of networks: popular networks like ResNet [42], VGG [43] and Inception [38], stacked U-Nets (SUNet) [44], dilated residual Table 3.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 2× · see citing works
“Such layers include language-modeling heads used during training in language processing networks (like BERT [37]) which are then replaced by task-specific heads during inference, auxiliary classifiers used in Inception networks [38] which are removed altogether for deployment, and the entirety of…”
MethodImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755 · cited 1× · see citing works
“Some of the lower-parameter networks (MobileNet v2 [48], SqueezeNet [49], MNAS-Net [50], ShuffleNet v2 [51], and EfficientNet [52]) do not fully recover accuracy when applying the basic workflow.”
BackgroundEfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · cited 1× · see citing works
“Some of the lower-parameter networks (MobileNet v2 [48], SqueezeNet [49], MNASNet [50], ShuffleNet v2 [51], and EfficientNet [52]) do not fully recover accuracy when applying the basic workflow.”
BackgroundGANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365 · see citing works
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
Ma, N., Zhang, X., Zheng, H.-T., & Sun, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_8 · Cited by 7,356 · cited 1× · see citing works
“Some of the lower-parameter networks (MobileNet v2 [48], SqueezeNet [49], MNASNet [50], ShuffleNet v2 [51], and EfficientNet [52]) do not fully recover accuracy when applying the basic workflow.”
BackgroundSqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201 · cited 2× · see citing works
“Some of the lower-parameter networks (MobileNet v2 [48], SqueezeNet [49], MNASNet [50], ShuffleNet v2 [51], and EfficientNet [52]) do not fully recover accuracy when applying the basic workflow.”
BackgroundGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · see citing works
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-4009 · Cited by 3,538 · cited 1× · see citing works
“To study the behavior of our workflow on NLP tasks, we select recurrent-based translation network (GNMT [58]), Transformer-based translation network (FairSeq Trans-former [59]), and two language modeling networks (Transformer-XL [60] and BERT [37]).”
MethodMnasNet: Platform-Aware Neural Architecture Search for Mobile
Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., & Le, Q. V. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00293 · Cited by 3,416 · cited 1× · see citing works
“Some of the lower-parameter networks (MobileNet v2 [48], SqueezeNet [49], MNASNet [50], ShuffleNet v2 [51], and EfficientNet [52]) do not fully recover accuracy when applying the basic workflow.”
BackgroundChannel Pruning for Accelerating Very Deep Neural Networks
He, Y., Zhang, X., & Sun, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.155 · Cited by 2,647 · cited 5× · see citing works
“Furthermore, the pattern used to prune models may adhere to a specific structure [10,17–19] or follow no structure at all [7,11].”
BackgroundMethodInfluentialYu, F., Koltun, V., & Funkhouser, T. A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.75 · Cited by 1,925 · cited 1× · see citing works
“[45].”
Learning Structured Sparsity in Deep Neural NetworksPreprint
Wen, W. X., Wu, C., Wang, Y., Chen, Y., & Li, H. H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.03665 · Cited by 1,861 · cited 2× · see citing works
“However, at higher levels of sparsity these pruning methods lose model accuracy [10,20–23].”
Pruning Convolutional Neural Networks for Resource Efficient InferencePreprint
Molchanov, P. A., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.06440 · Cited by 1,487 · see citing works
Rethinking the Value of Network PruningPreprint
Liu, Z., Sun, M., Zhou, T., Huang, G. M., & Darrell, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.05270 · Cited by 1,225 · see citing works
Exploring the Limits of Weakly Supervised Pretraining
Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., & van der Maaten, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_12 · Cited by 1,224 · see citing works
Importance Estimation for Neural Network Pruning
Molchanov, P., Mallya, A., Tyree, S., Frosio, I., & Kautz, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01152 · Cited by 983 · cited 1× · see citing works
“Pruning methods typically eliminate weights using weight magnitude based metrics [7] or some salience-based criteria [15]).”
MethodTransformer-XL: Attentive Language Models Beyond a Fixed-Length ContextPreprint
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019) · arXiv (Cornell University) · doi:10.4230/lipics.ecrts.2023.7 · Cited by 860 · see citing works
To prune, or not to prune: exploring the efficacy of pruning for model compression
Zhu, M., & Gupta, S. (2017) · arXiv · doi:10.48550/arxiv.1710.01878 · Cited by 805 · cited 1× · see citing works
“Iterative methods prune weights gradually over a number of steps, in each step eliminating either a fixed number of weights [6, 7] or choosing a fraction of weights based on an analytical function [9].”
MethodStructured Pruning of Deep Convolutional Neural Networks
Anwar, S., Hwang, K., & Sung, W. (2017) · ACM Journal on Emerging Technologies in Computing Systems · doi:10.1145/3005348 · Cited by 709 · cited 1× · see citing works
“Furthermore, the pattern used to prune models may adhere to a specific structure [10,17–19] or follow no structure at all [7,11].”
MethodSNIP: Single-shot Network Pruning based on Connection Sensitivity
Lee, N., Ajanthan, T., & Torr, P. H. S. (2018) · arXiv · doi:10.48550/arxiv.1810.02340 · Cited by 685 · cited 1× · see citing works
“A variety of pruning schedules have been proposed – single step [16] and gradual/iterative [10,11,17].”
BackgroundSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · arXiv · doi:10.48550/arxiv.1606.05250 · Cited by 678 · see citing works
Mesh-TensorFlow: Deep Learning for SupercomputersPreprint
Shazeer, N., Cheng, Y.-L., Parmar, N., Tran, D., Vaswani, A., Koanantakool, P., Hawkins, P. G., Lee, H., Hong, M., Young, C., et al. (2018) · Neural Information Processing Systems · doi:10.48550/arxiv.1811.02084 · Cited by 452 · see citing works
Improving Semantic Segmentation via Video Propagation and Label Relaxation
Zhu, Y., Sapra, K., Reda, F. A., Shih, K. J., Newsam, S. D., Tao, A., & Catanzaro, B. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00906 · Cited by 429 · cited 1× · see citing works
“To study object detection and segmentation, we use networks from PyTorch Torchvision, Detectron2 [53], NVIDIA Deep Learning Examples for Tensor Cores [54], and NVIDIA ADLR [55] repositories.”
MethodBillion-scale semi-supervised learning for image classificationPreprint
Yalnız, İ. Z., Jeǵou, H., Chen, K., Paluri, M., & Mahajan, D. K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.00546 · Cited by 387 · cited 1× · see citing works
“We also examine networks trained with weakly (WSL) or semiweakly supervised learning (SWSL) methods [46, 47] which use additional data to improve accuracy.”
MethodRigging the Lottery: Making All Tickets WinnersPreprint
Evci, U., Gale, T., Menick, J., Castro, P. S., & Elsen, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.11134 · Cited by 290 · see citing works
Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
Wu, H., Judd, P., Zhang, X., Isaev, M., & Micikevicius, P. (2020) · arXiv · doi:10.48550/arxiv.2004.09602 · Cited by 246 · cited 4× · see citing works
“After fine-tuning, the networks are quantized to INT8 using max calibration [41].”
MethodInfluentialSelf-citationExploring Sparsity in Recurrent Neural NetworksPreprint
Narang, S., Elsen, E., Diamos, G., & Sengupta, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.05119 · Cited by 218 · cited 2× · see citing works
“Sparse models can also be obtained by starting from randomly-initialized dense models and gradually introducing sparsity as training progresses [9,21,26–31].”
BackgroundInfluentialExploring the Granularity of Sparsity in Convolutional Neural Networks
Mao, H., Han, S., Pool, J., Li, W., Liu, X., Wang, Y., & Dally, W. J. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.241 · Cited by 201 · see citing works
Self-citationZhu, M., Zhang, T., Gu, Z., & Xie, Y. (2019) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3352460.3358269 · Cited by 150 · see citing works
Learning N:M Fine-grained Structured Sparse Neural Networks From ScratchPreprint
Zhou, A., Ma, Y., Zhu, J., Liu, J., Zhang, Z., Yuan, K., Sun, W., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.04010 · Cited by 114 · cited 1× · see citing works
“For this, one may need to train with dynamic sparsity masks, evolving them during training [65] (as opposed to static masks computed once during pruning, which this paper shows to suffice for inference).”
BackgroundElsen, E., Dukhan, M., Gale, T., & Simonyan, K. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01464 · Cited by 111 · see citing works
The Lottery Ticket Hypothesis: Training Pruned Neural Networks.
Frankle, J., & Carbin, M. (2018) · Cited by 106 · see citing works
Filter Sketch for Network Pruning
Lin, M., Cao, L., Li, S., Ye, Q., Tian, Y., Liu, J., Tian, Q. C., & Ji, R. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3084206 · Cited by 103 · see citing works
Block-Sparse Recurrent Neural NetworksPreprint
Narang, S., Undersander, E., & Diamos, G. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02782 · Cited by 99 · see citing works
Accelerator-Aware Pruning for Convolutional Neural Networks
Kang, H. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2911674 · Cited by 90 · cited 2× · see citing works
“Additionally, there has been work investigating fine-grained structured sparsity and motivating the need to prune in a fine-grained pattern that is conducive to hardware acceleration [17, 33].”
BackgroundMethodBalanced Sparsity for Efficient DNN Inference on GPU
Yao, Z., Cao, S., Xiao, W., Zhang, C., & Nie, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015676 · Cited by 83 · cited 3× · see citing works
“Additionally, there has been work investigating fine-grained structured sparsity and motivating the need to prune in a fine-grained pattern that is conducive to hardware acceleration [17, 33].”
BackgroundMethodThe Lottery Ticket Hypothesis at Scale
Frankle, J., Dziugaite, G. K., Roy, D. M., & Carbin, M. (2019) · Cited by 71 · see citing works
Hestness, J., Ardalani, N., & Diamos, G. (2019) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3293883.3295710 · Cited by 70 · cited 1× · see citing works
“For example, metadata for the first row of matrix in Figure 1 is [[0, 3], [1, 2]].”
MethodDSD: Regularizing Deep Neural Networks with Dense-Sparse-Dense Training Flow.Preprint
Han, S., Pool, J., Narang, S., Mao, H., Tang, S., Elsen, E., Catanzaro, B., Tran, J. J., & Dally, W. J. (2016) · arXiv (Cornell University) · Cited by 64 · see citing works
Self-citationStacked U-Nets: A No-Frills Approach to Natural Image SegmentationPreprint
Shah, S., Ghosh, P., Davis, L. S., & Goldstein, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.10343 · Cited by 42 · cited 1× · see citing works
“Table 2 shows the accuracy of a wide variety of networks: popular networks like ResNet [42], VGG [43] and Inception [38], stacked U-Nets (SUNet) [44], dilated residual Table 3.”
BackgroundAccelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable MasksPreprint
Hubara, I., Chmiel, B., Island, M., Banner, R., Naor, J. S., & Soudry, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08124 · Cited by 40 · see citing works
Sparse connection and pruning in large dynamic artificial neural networks
Ström, N. (1997) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1997-708 · Cited by 39 · see citing works
SQuantizer: Simultaneous Learning for Both Sparse and Low-precision Neural Networks
Park, M. S., Xu, X., & Brick, C. (2018) · arXiv · doi:10.48550/arxiv.1812.08301 · Cited by 8 · see citing works
Campfire: Compressible, Regularization-Free, Structured Sparse Training for Hardware AcceleratorsPreprint
Gamboa, N., Kudrolli, K., Dhoot, A., & Pedram, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.03253 · Cited by 8 · see citing works
Self-Supervised Generative Adversarial Compression.
Yu, C., & Pool, J. (2020) · neural information processing systems · Cited by 6 · see citing works
Self-citationStructured Pruning for Efficient ConvNets via Incremental RegularizationPreprint
Wang, H., Zhang, Q., Wang, Y., & Hu, H. R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.08390 · Cited by 6 · see citing works
Flagged first, then by citations
3 of the 52 references couldn’t be matched to a known paper.
Research works printed but not linked: 8
- [1]1. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. CoRR , abs/1712.00409, 2017. · arXiv
- [3]3. Md. Mostofa Ali Patwary, Milind Chabbi, Heewoo Jun, Jiaji Huang, Gregory Frederick Diamos, and Kenneth Church. Language modeling at scale. CoRR , abs/1810.10045, 2018. · arXiv
- [7]7. Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems , pages 1135-1143, 2015.
- [25]25. Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing fine-tuning and rewinding in neural network pruning. In ICLR , 2020.
- [30]30. Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, A¨ aron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. CoRR , abs/1802.08435, 2018. · arXiv
- [35]35. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS , 2020.
- [38]38. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. CoRR , abs/1409.4842, 2014. · arXiv
- [40]40. Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ ar, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR , abs/1405.0312, 2014. · arXiv
Cited by
Sorted by citations
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154 · see citing works
Wang, Y., Zhang, C., Xie, Z., Guo, C., Liu, Y., & Leng, J. (2021) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca52012.2021.00088 · Cited by 113 · see citing works
“Sparse Tensor Core [42], [45], [72] only leverages weight sparsity and the input remains dense, which requires only dense im2col.”
BackgroundMethodInfluentialSurvey of Different Large Language Model Architectures: Trends, Benchmarks, and Challenges
Shao, M., Basit, A., Karri, R., & Shafique, M. (2024) · IEEE Access · doi:10.1109/access.2024.3482107 · Cited by 110 · see citing works
An Algorithm–Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers
Fang, C., Zhou, A., & Wang, Z. (2022) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2022.3197282 · Cited by 94 · see citing works
“In addition, in the forward pass, we leverage the popular groupwise magnitude pruning [7], [9].”
MethodInfluentialOn Efficient Training of Large-Scale Deep Learning Models
Shen, L., Sun, Y., Yu, Z., Ding, L., Tian, X., & Tao, D. (2024) · ACM Computing Surveys · doi:10.1145/3700439 · Cited by 126 · see citing works
“%) sparsity pattern [130], enabling twice the mathematical throughput of dense math.”
A Survey on Efficient Training of Transformers
Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., & Shen, C. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/764 · Cited by 78 · see citing works
A review of AI edge devices and lightweight CNN and LLM deployment
Sun, K., Wang, X., Miao, X., & Zhao, Q. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128791 · Cited by 76 · see citing works
Global Vision Transformer Pruning with Hessian-Aware Saliency
Yang, H., Yin, H., Shen, M., Molchanov, P. A., Li, H. H., & Kautz, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01779 · Cited by 70 · see citing works
“This comes with a limitation of requiring the input and output dimensions of all linear projections to be divisible by 16 [27].”
BackgroundA Survey on Model Compression and Acceleration for Pretrained Language Models
Xu, C., & McAuley, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26255 · Cited by 70 · see citing works
1xN Pattern for Pruning Convolutional Neural Networks
Lin, M., Zhang, Y., Li, Y., Chen, B., Chao, F., Wang, M., Li, S., Tian, Y., & Ji, R. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3195774 · Cited by 54 · see citing works
“At the second pruning step, inspired by [10] which conducts channel permutations to preserve more high-magnitude weights in the N:M weight pruning, we realize that the layout of V can be altered to further relieve the pruning impact.”
InfluentialFluctuation-Based Adaptive Structured Pruning for Large Language Models
An, Y., Xu, Z., Yu, C., Tang, M., & Wang, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i10.28960 · Cited by 60 · see citing works
Self-citationAdvances in Pruning and Quantization for Natural Language Processing
Bibi, U., Mazhar, M., Sabir, D., Butt, M. F. U., Hassan, A., Ghazanfar, M. A., Khan, A. A., & Abdul, W. (2024) · IEEE Access · doi:10.1109/access.2024.3465631 · Cited by 47 · see citing works
“The work in [36] concerns the collaboration of 50% fine-grained structural sparsity and 4-bits down quantization of transformer backbones concerning BERT ensuring the accuracy of the model with 16 times compression.”
BackgroundOn-device Training: A First Overview on Existing Systems
Zhu, S., Voigt, T., Rahimian, F., & Ko, J. (2024) · ACM Transactions on Sensor Networks · doi:10.1145/3696003 · Cited by 47 · see citing works
“The most common hardware accelerators are graphics processing units (GPUs) [86] and tensor processing units (TPUs) [51].”
VEGETA: Vertically-Integrated Extensions for Sparse/Dense GEMM Tile Acceleration on CPUs
Jeong, G., Damani, S., Bambhaniya, A. R., Qin, E., Hughes, C. J., Subramoney, S., Kim, H., & Krishna, T. (2023) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071058 · Cited by 36 · see citing works
“For example, edge devices with tight area/power budget [46] prefer CPUs since GPUs/accelerators cannot be easily deployed in addition to CPUs.”
BackgroundMethodInfluentialBoost Vision Transformer with GPU-Friendly Sparsity and Quantization
Yu, C., Chen, T., Gan, Z., & Fan, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02170 · Cited by 33 · see citing works
“The coarse-grained sparsity has a regular sparse pattern which can facilitate acceleration with algebra libraries [33].”
BackgroundMethodInfluentialSelf-citationRGP: Neural Network Pruning Through Regular Graph With Edges Swapping
Chen, Z., Xiang, J., Lu, Y., Xuan, Q., Wang, Z., Chen, G., & Yang, X. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3280899 · Cited by 29 · see citing works
“Similarly, Mishra et al. [30] show a design of NVIDIA ampere GPU architecture for sparsity support, and introduce a 2:4 sparsity pattern of block pruning.”
MethodReal-Time Semantic Segmentation: A brief survey and comparative study in remote sensing
Broni-Bediako, C., Xia, J., & Yokoya, N. (2023) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2023.3321258 · Cited by 30 · see citing works
“These techniques are used to remove “unimportant” weights and neurons in a model to reduce its number of computation arithmetic operations and make it smaller [70], [71], [72], [73] [see Figure 2(g) and (h)].”
MethodNViT: Vision Transformer Compression and Parameter Redistribution.Preprint
Yang, H., Yin, H., Molchanov, P. A., Li, H., & Kautz, J. (2021) · arXiv (Cornell University) · Cited by 28 · see citing works
Plug-and-Play: An Efficient Post-training Pruning Method for Large Language ModelsPreprint
Zhang, Y., Bai, H., Lin, H., Zhao, J., Hou, L., & Cannistraci, C. V. (2024) · Preprints.org · doi:10.20944/preprints202310.1487.v2 · Cited by 27 · see citing works
Efficient N:M Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Design
Fang, C., Sun, W., Zhou, A., & Wang, Z. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3317789 · Cited by 25 · see citing works
“In recent years, there has been a growing interest in leveraging fine-grained structured sparsity [21], [22], [23], [24], [25] for efficient DNN acceleration.”
BackgroundMethodInfluentialBBS: Bi-Directional Bit-Level Sparsity for Deep Learning Acceleration
Chen, Y., Meng, J., Seo, J.-S., & Abdelfattah, M. S. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00048 · Cited by 25 · see citing works
“Numerous efficiency algorithms [21], [30], [31] and hardware prototypes [6], [13], [14], [42], [43] have been proposed to leverage value-based sparsity in DNNs to reduce the cost of storing and deploying DNNs.”
BackgroundHow Well Do Sparse ImageNet Models Transfer?
Iofinova, E., Peste, A., Kurtz, M., & Alistarh, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01195 · Cited by 22 · see citing works
“at Thanks to this trend, a set of fairly consistent accuracy benchmarks has emerged for pruning, along with increasingly efficient computational support [11, 20, 40, 52].”
BackgroundRED++ : Data-Free Pruning of Deep Neural Networks via Input Splitting and Output Merging
Yvinec, E., Dapogny, A., Cord, M., & Bailly, K. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3179616 · Cited by 23 · see citing works
“4% upon AccS [46] on EfficientNet B0 while preserving the accuracy.”
BackgroundMethodInfluentialAttentive Fine-Grained Structured Sparsity for Image Restoration
Oh, J., Kim, H., Nah, S., Hong, C., Choi, J., & Lee, K. M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01715 · Cited by 21 · see citing works
“Such sparsity constraint has the potential of hardware acceleration, where 2:4 sparsity pattern has recently been supported in NVIDIA Ampere generation GPUs [37].”
BackgroundMethodInfluentialPost-Training Sparsity-Aware QuantizationPreprint
Shomron, G., Gabbay, F., Kurzum, S., & Weiser, U. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.11010 · Cited by 21 · see citing works
“NVIDIA also recently introduced weight sparsity acceleration in its Ampere microarchitecture [17, 19].”
MethodLuo, X., Liu, D., Kong, H., Huai, S., Chen, H., Xiong, G., & Liu, W. (2024) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3701728 · Cited by 24 · see citing works
ELSA: Exploiting Layer-wise N:M Sparsity for Vision Transformer Acceleration
Huang, N.-C., Chang, C.-C., Lin, W., Taka, E., Marculescu, D., & Wu, K. H. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00799 · Cited by 18 · see citing works
“…structured pruning methods [23, 34, 14] remove rows, columns, or N:M sparsity Given N & M Find a better sparse mask [15, 38] Training algorithm [24, 39, 21] Find N & M [31] Figure 1: Three categories of methodologies for N : M semi-structured pruning channels that do not significantly impact…”
BackgroundMethodInfluentialRegister Tiling for Unstructured Sparsity in Neural Network Inference
Wilkinson, L., Cheshmi, K., & Dehnavi, M. M. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3591302 · Cited by 17 · see citing works
“To turn FLOP reduction into performance, semi-structured or pattern-based pruning methods are proposed [Guo et al. 2020; Huang et al. 2022; Lagunas et al. 2021; Mao et al. 2017; Mishra et al. 2021; Zhu et al. 2019], these patterns are better fit to current computing hardware and software systems.”
MethodERA-BS: Boosting the Efficiency of ReRAM-Based PIM Accelerator With Fine-Grained Bit-Level Sparsity
Liu, F., Zhao, W., Wang, Z., Chen, Y., Liang, X., & Jiang, L. (2023) · IEEE Transactions on Computers · doi:10.1109/tc.2023.3290869 · Cited by 15 · see citing works
“In addition, NVIDIA implements 2/4 weight sparsity in its commercial A100 GPUs [51].”
MethodInvestigating Hallucinations in Pruned Large Language Models for Abstractive Summarization
Chrysostomou, G., Zhao, Z., Williams, M., & Αλέτρας, Ν. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00695 · Cited by 18 · see citing works
Soft Masking for Cost-Constrained Channel Pruning
Humble, R., Shen, M., Latorre, J. A., Darve, E., & Álvarez, J. M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20083-0_38 · Cited by 15 · see citing works
Self-citationHigh-Performance Tensor Learning Primitives Using GPU Tensor Cores
Liu, X.-Y., Zhang, Z., Wang, Z., et al. (2022) · IEEE Transactions on Computers · doi:10.1109/tc.2022.3222955 · Cited by 13 · see citing works
“GPU tensor cores [31] are specialized hardware that employs mixed-precision computation [32] to accelerate tensor operations and have been studied in many recent works, such as accelerating neural networks [33][34], sci-entific computing [35][36] and quantum-based molecular dynamics simulations [37].”
MethodAC/DC: Alternating Compressed/DeCompressed Training of Deep Neural NetworksPreprint
Peşte, A., Iofinova, E., Vladu, A., & Alistarh, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.12379 · Cited by 13 · see citing works
“To this end, entire families of model compression methods have been developed, such as pruning [29] and quantization [22], which are now accompanied by hardware and software support [55, 8, 11, 43, 23].”
BackgroundMethodTwo sparsities are better than one: unlocking the performance benefits of sparse–sparse networks
Hunter, K., Spracklen, L., & Ahmad, S. (2022) · Neuromorphic Computing and Engineering · doi:10.1088/2634-4386/ac7c8a · Cited by 12 · see citing works
“Hardware platforms with dedicated logic for exploiting sparsity have begun to appear [58], but the performance gains remain modest [48].”
BackgroundDSTC: Dual-Side Sparse Tensor Core for DNNs Acceleration on Modern GPU Architectures
Zhang, C., Wang, Y., Xie, Z., Guo, C., Liu, Y., Leng, J., Ji, Z., Xie, Y., & Huang, R. (2024) · IEEE Transactions on Computers · doi:10.1109/tc.2024.3475814 · Cited by 11 · see citing works
“Notably, NVIDIA’s latest Ampere GPU introduces Sparse Tensor Cores, embracing fine-grained structural pruning [5], [22], [24], marking a significant milestone in GPU design.”
InfluentialBalanced Column-Wise Block Pruning for Maximizing GPU Parallelism
Park, C., Park, M.-C., Oh, H. J., et al. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i8.26126 · Cited by 11 · see citing works
“Comparison with BLP The pruning unit size of BCBP is a R 32 × 1 .”
BackgroundMethodInfluentialBias in Pruned Vision Models: In-Depth Analysis and Countermeasures
Iofinova, E., Peşte, A., & Alistarh, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02334 · Cited by 10 · see citing works
“We additionally prune to N:M (2:4, 1:4 and 1:8) sparsity patterns [44] in Appendix F, with similar results to lower-sparsity models pruned without this restriction.”
MethodResultThe Cultivated Practices of Text-to-Image Generation
Oppenlaender, J. (2024) · Springer eBooks · doi:10.1007/978-3-031-66528-8_14 · Cited by 12 · see citing works
Duan, C., Yang, J., He, X., Qi, Y., Wang, Y., Wang, Y., He, Z., Yan, B., Wang, X., Jia, X., et al. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3330819 · Cited by 10 · see citing works
“We choose the well-known fine-grained 2:4 structured pruning algorithm proposed by NVIDIA to verify the orthogonal nature of our algorithm relative to traditional pruning techniques [39].”
MethodTB-STC: Transposable Block-wise N:M Structured Sparse Tensor Core
Liu, J., Zeng, S., Zhao, J., Ding, L., Wang, Z., Li, J., Zhu, Z., Ning, X., Zhang, C., Wang, Y., et al. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00075 · Cited by 10 · see citing works
“It means each tile with M elements has at most N non-zeros [42].”
InfluentialSASCHA—Sparsity-Aware Stochastic Computing Hardware Architecture for Neural Network Acceleration
Romaszkan, W., Li, T., & Gupta, P. (2022) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2022.3197503 · Cited by 9 · see citing works
“Because of that, constrained, or structured, pruning has been introduced, including filter-wise [18], group-wise [49], and pattern sparsity [31].”
BackgroundCRISP: Hybrid Structured Sparsity for Class-Aware Model Pruning
Aggarwal, S., Binici, K., & Mitra, T. (2024) · doi:10.23919/date58400.2024.10546782 · Cited by 9 · see citing works
Are Straight-Through gradients and Soft-Thresholding all you need for Sparse Training?
Vanderschueren, A., & Vleeschouwer, C. D. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00380 · Cited by 9 · see citing works
“While zeroing the weights in a structured manner might lead to higher acceleration when considering specific hardware implementations, the accuracy obtained with unstructured sparsity is generally higher at a given sparsity level[24, 12, 37], and significant speed up can be achieved with unstructured sparsity on off-t…”
BackgroundWang, J., Moreira, J., Cao, Y., & Gopaluni, R. B. (2023) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2023.10.503 · Cited by 8 · see citing works
“The sparse connections can also help to save computational time costs and memory storage (Kepner et al., 2018; Mishra et al., 2021; Kepner et al., 2019).”
BackgroundOptimizing Structured-Sparse Matrix Multiplication in RISC-V Vector Processors
Titopoulos, V., Alexandridis, K., Peltekis, C., et al. (2025) · IEEE Transactions on Computers · doi:10.1109/tc.2025.3533083 · Cited by 7 · see citing works
“In most practical applications [4], [8], blocks are small and N : M sparsity patterns of 1:2, 1:4 or 2:4 are supported, where each block of M elements may contain up to N non-zero elements.”
InfluentialBoost Transformer-based Language Models with GPU-Friendly Sparsity and Quantization
Yu, C. H., Chen, T., & Gan, Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.15 · Cited by 7 · see citing works
Self-citationIterative Soft Shrinkage Learning for Efficient Image Super-Resolution
Wang, J., Wang, H., Zhang, Y., Fu, Y., & Tao, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01157 · Cited by 7 · see citing works
“Besides compared methods, another emerging trend of weight pruning is to employ the N:M sparsity [14, 25].”
BackgroundMethodEfficient Memory Integration: MRAM-SRAM Hybrid Accelerator for Sparse On-Device Learning
Zhang, F., Sridharan, A., Tsai, W., et al. (2024) · Proceedings - ACM IEEE Design Automation Conference · doi:10.1145/3649329.3657390 · Cited by 7 · see citing works
Bandwidth-Efficient Sparse Matrix Multiplier Architecture for Deep Neural Networks on FPGA
Mahesh, M., Nalesh, S., & Kala, S. (2021) · IEEE International System-on-Chip Conference (SOCC) · doi:10.1109/socc52499.2021.9739346 · Cited by 7 · see citing works
“GPU architectures are evolving to accelerate compressed DNNs, but they still require some kind or structure to the matrix, like Nvidia’s 2:4 sparsity pattern requirement in Ampere architecture [4].”
BackgroundScalable Deep Learning-Based Microarchitecture Simulation on GPUs
Pandey, S., Li, L., Flynn, T., et al. (2022) · 2012 International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc41404.2022.00084 · Cited by 7 · see citing works
“Tensor Core can provide nearly 2× speedup for matrix multiplication with 2:4 sparsity [30].”
MethodStarSPA: Stride-Aware Sparsity Compression for Efficient CNN Acceleration
Pham, N.-S., Shin, S., Xu, L., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3353313 · Cited by 7 · see citing works
“However, this approach may come at the cost of a more significant compromise in accuracy compared to unstructured sparsity [29], [30], [31].”
BackgroundSparseMAE: Sparse Training Meets Masked Autoencoders
Zhou, A., Yang, L., Qin, Z., Liu, J., Pan, J., Zhang, R., Zhao, R., Gao, P., & Li, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01482 · Cited by 7 · see citing works
“We conduct experiments with two different granularities for sparse training, consisting of unstructured sparsity [14, 21] and hardware-friendly N:M sparsity [28, 42].”
MethodEvaluating performance of AI operators using roofline model
Chen, Z., Zheng, F., Yu, Q., Sun, R., Guo, F., & Chen, Z.-N. (2021) · Applied intelligence (Boston) · doi:10.1007/s10489-021-02794-5 · Cited by 5 · see citing works
Park, T., Kang, S., Jang, M.-H., et al. (2023) · Proceedings - International Conference on Data Engineering · doi:10.1109/icde55515.2023.00189 · Cited by 6 · see citing works
DeepAAT: Deep Automated Aerial Triangulation for Fast UAV-based mapping
Chen, Z., Li, J., Li, Q., et al. (2024) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2024.104190 · Cited by 6 · see citing works
Taylor-Series-Expansion-Based Vision Transformer Models
Yu, C., Chen, T., & Gan, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3578827 · Cited by 5 · see citing works
“[51] leverage the combination of sparse patterns and the sparse acceleration instructions [26], [27] provided by GPU platforms to boost deployment efficiency.”
Self-citationHardware Acceleration of SpMV Multiplier for Deep Learning
Mahesh, M. N., Nalesh, S., & Kala, S. (2021) · International Symposium on VLSI Design and Test (VDAT) · doi:10.1109/vdat53777.2021.9600988 · Cited by 4 · see citing works
Kim, D., & Kim, J. (2022) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc55196.2022.9952814 · Cited by 4 · see citing works
“%) sparsity pattern, i.e., out of 4 elements, at least 2 elements are zero [3].”
BackgroundDeMM: A Decoupled Matrix Multiplication Engine Supporting Relaxed Structured Sparsity
Peltekis, C., Titopoulos, V., Nicopoulos, C. A., et al. (2024) · IEEE Computer Architecture Letters · doi:10.1109/lca.2024.3355178 · Cited by 4 · see citing works
“In most practical applications [4], [6], [7], blocks are small and fine-grained N : M sparsity patterns of 1:2, 1:4 or 2:4 are supported, where each block of M elements may contain up to N non-zero elements.”
BackgroundMethodFruit Image Classification using the Inception-V3 Deep Learning Model
Sunday, S. E., Ji, R., Abdalla, A. N., et al. (2023) · doi:10.1109/iccd59681.2023.10420760 · Cited by 4 · see citing works
STuning-DL: Model-Driven Autotuning of Sparse GPU Kernels for Deep Learning
Castro, R. L., Andrade, D., & Fraguela, B. B. (2024) · IEEE Access · doi:10.1109/access.2024.3402326 · Cited by 4 · see citing works
“D blocks of elements of length l ; N:M pruning [35] divides the matrix into vectors of consecutive M elements and selects the best N out of M for each one of them; V:N:M pruning [5] divides the matrix into blocks of size V × M , drops a subset of M − 4 columns for each group, and selects N elements…”
BackgroundError Checking for Sparse Systolic Tensor Arrays
Peltekis, C., Filippas, D., & Dimitrakopoulos, G. (2024) · doi:10.1109/aicas59952.2024.10595917 · Cited by 4 · see citing works
Towards Fully Sparse Training: Information Restoration with Spatial Similarity
Xu, W., He, X., Cheng, K., et al. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20198 · Cited by 4 · see citing works
“The former depends on a well-trained dense model and needs further retraining (Han et al. 2015; He et al. 2019b; Mishra et al. 2021), which is not the scope of this work.”
BackgroundMethodInfluentialPupil Detection for Augmented and Virtual Reality based on Images with Reduced Bit Depths
Fiala, G., Ye, Z., & Steger, C. (2022) · IEEE Sensors Applications Symposium (SAS) · doi:10.1109/sas54819.2022.9881378 · Cited by 4 · see citing works
High Performance and Power Efficient Accelerator for Cloud Inference
Yao, J., Zhou, H., Zhang, Y., Li, Y., Feng, C., Chen, S., Chen, J., Wang, Y., & Hu, Q. (2023) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10070941 · Cited by 4 · see citing works
“Computations with low arithmetic intensities, such as sparse embedding, can be bottle-necked due to memory bandwidth when large-capacity on-chip memory fails to accommodate the entire data required [40], [61], [68].”
BackgroundFinkbeiner, J., Gmeinder, T., Pupilli, M., Titterton, A., & Neftci, E. O. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i11.29087 · Cited by 4 · see citing works
Zhang, Y., Zhao, J., Wu, W., Muscoloni, A., & Cannistraci, C. V. (2024) · Preprints.org · doi:10.20944/preprints202207.0139.v4 · Cited by 4 · see citing works
The Impact of Uniform Inputs on Activation Sparsity and Energy-Latency Attacks in Computer Vision
Müller, A., & Quiring, E. (2024) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw63631.2024.00016 · Cited by 3 · see citing works
“For example, NVIDIA’s A100 GPU can accelerate 2:4 sparsity if every 2 out of 4 weights in a matrix are zero [26].”
MethodWang, H., Ding, Y., Liu, Y., et al. (2024) · IEEE Transactions on Computers · doi:10.1109/tc.2024.3517745 · Cited by 3 · see citing works
“These techniques include random sparsity, Non-2:4 [25], plain 2:4 [6], [15], channel permutation [16], Layer-wise sparsity [26], block sparsity [9], and FS-AMM.”
MaxQ: Multi-Axis Query for N:m Sparsity Network
Xiang, J., Li, S., Chen, J., Chen, Z., Huang, T., Peng, L., & Liu, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01500 · Cited by 3 · see citing works
“According to the granularity of sparse weights, network sparsity can be categorized into structured sparsity [19, 30, 41], unstructured sparsity [10, 11, 28] and semi-structured sparsity [5, 9, 31, 35].”
BackgroundMethodInfluentialEfficient Edge AI for Next Generation Smart Mirror Applications
Kucza, N., Porrmann, F., Stollenwerk, C., & Hagemeyer, J. (2025) · IEEE Access · doi:10.1109/access.2025.3574492 · Cited by 3 · see citing works
“CRISP leverages a hybrid structured sparsity pattern that combines fine-grained N:M sparsity with coarse-grained block sparsity [24] to achieve high compression rates while maintaining model accuracy.”
A Slice and Dice Approach to Accelerate Compound Sparse Attention on GPU
Li, H., Choi, J., & Ahn, J. H. (2022) · 2022 IEEE International Symposium on Workload Characterization (IISWC) · doi:10.1109/iiswc55918.2022.00019 · Cited by 3 · see citing works
“Starting from the Ampere and Hopper architecture GPUs, NVIDIA introduced sparse tensor cores [22] to accelerate the sparse operation of the 2:4 (50%) fine-grained structured patterns.”
MethodDeriving Coding-Specific Sub-Models from LLMs using Resource-Efficient Pruning
Puccioni, L., Farshin, A., Scazzariello, M., Wang, C., Chiesa, M., & Kostić, D. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00028 · Cited by 2 · see citing works
“Although our experiments primarily focuses on 50% unstructured sparsity, our takeaways are also applicable to structured pruning ( e.g., structured N:M sparsity [55]), as shown by Wanda [14].”
Zero-shot Sharpness-Aware Quantization for Pre-trained Language Models
Zhu, M., Zhong, Q., Shen, L., Ding, L., Liu, J., Du, B., & Tao, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.696 · Cited by 2 · see citing works
Zhang, Y., Zhao, J., Wu, W., Muscoloni, A., & Cannistraci, C. V. (2023) · Preprints.org · doi:10.20944/preprints202207.0139.v3 · Cited by 2 · see citing works
Sparsity Limit to Prune Large Language Models for on-Device AI Assistants: Llama-2 as an ExamplePreprint
Liu, B., & Xu, Y. (2024) · Preprints.org · doi:10.20944/preprints202407.1568.v2 · Cited by 2 · see citing works
POEM: Pruning with Output Error Minimization for Compressing Deep Neural Networks
KAMMA, K., & Wada, T. (2024) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2023edp7165 · Cited by 1 · see citing works
“Even though the unstructured pruning makes an irregular structure of the weight tensor, the computation environments dedicated for such sparse models are now available [42].”
Yan, Y., & Li, X. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111831 · Cited by 1 · see citing works
Sparsity-Aware Hardware: From Overheads to Performance Benefits
Shi, M., Kneip, A., Chauvaux, N., Sun, J., Frenkel, C., & Verhelst, M. (2025) · IEEE Solid-State Circuits Magazine · doi:10.1109/mssc.2025.3549709 · Cited by 1 · see citing works
Periodic Online Testing for Sparse Systolic Tensor Arrays
Peltekis, C., Nicopoulos, C. A., & Dimitrakopoulos, G. (2025) · International Conference on Modern Circuits and Systems Technologies · doi:10.1109/mocast65744.2025.11083733 · Cited by 1 · see citing works
“On the contrary, structured sparsity [3], [4] alleviates the meta-data cost and complexity by enforcing a constraint on the maximum possible number of non-zero elements present in every fixed-size block of consecutive input elements.”
Efficient One-Shot Pruning of Large Language Models with Low-Rank Approximation
Xu, Y., Cao, C., Yuan, F., Mi, R., Sun, N., Wang, D., & Liu, Y. (2024) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc54092.2024.10831406 · Cited by 1 · see citing works
“This indicates that sustaining a balanced pruning rate across output features is essential for effectively pruning LLMs. 3) Extension to Semi-Structured Sparsity: While we have discussed only unstructured pruning so far, our method is also easily adaptable to semi-structured N:M patterns [33].”
GEMMHeal: An Efficient Self-repairing Architecture for Matrix Multiplication
Qiao, Y., Liu, C., Zhou, H., et al. (2025) · International Symposium of Electronics Design Automation (ISEDA) · doi:10.1109/iseda65950.2025.11100433 · Cited by 1 · see citing works
Towards High Performance and Accurate BNN Inference on FPGA with Structured Fine-Grained Pruning
Fu, K., Qi, Z., Cai, J., et al. (2022) · doi:10.1145/3508352.3549368 · Cited by 1 · see citing works
Search Spaces for Neural Model TrainingPreprint
Stošić, D., & Stošić, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.12920 · Cited by 1 · see citing works
“We then show this methodology achieves competitive results on dozens of deep learning workloads, even when satisfying constraints needed to accelerate training and inference using Sparse Tensor Cores [24] in NVIDIA GPUs.”
BackgroundMethodInfluentialAn Inference Accelerator Design for Sparse Convolution Neural Network
Wang, D., Wu, T., Tan, H., & Li, H. (2021) · International Conference on Computer Science, Electronic Information Engineering and Intelligent Control Technology (CEI) · doi:10.1109/cei52496.2021.9574454
Hardware-aware design, search, and optimization of deep neural networks
Chitty-Venkata, S. S. (2023) · doi:10.31274/td-20240329-380
Sparsity Limit to Prune Large Language Models for on-Device AI Assistants: Llama-2 as an ExamplePreprint
Liu, B., & Xu, Y. (2024) · Preprints.org · doi:10.20944/preprints202407.1568.v1
Towards the Impact of Pruning Criteria and Tuning Approach on Model Interpretability and Quality
Shcherbin, A. (2024) · IEEE International Conference of Young Professionals in Electron Devices and Materials (EDM) · doi:10.1109/edm61683.2024.10615139
“On Nvidia GPUs model inference speed can be improved by semi-structured pruning that calls 2:4 sparsity [12].”
MethodSoft Pruning and Latent Space Dimensionality Reduction
Menzies, R., & Siebert, P. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651389
RSKCNN: introducing randomly-sparse-kernel CNN
Feng, Z., He, Y., & Li, Y. (2024) · doi:10.1117/12.3056706
Compression Strategies for Massive Natural Language Generation Models
Dayal, A., Angara, J., Gupta, S., & Saripalle, R. S. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-77081-4_7
Optimizing Deep Learning for Biomedical Imaging
Chaturvedi, A., Cao, G., & Feng, W.-C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-82768-6_4
“In the forward pass, tensor cores perform matrix-multiply-add (MMA) operations in TF32 format [20] for convolution, as shown in Fig.”
BackgroundMethodSNIPER Training: Single-Shot Sparse Training for Text-to-Speech
Lam, P., Zhang, H., Chen, N. F., Şişman, B., & Herremans, D. (2024) · IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon61640.2024.10902970
“The closest implementations currently available are NVIDIA Sparse Tensor Cores [9], where the inference speed doubles if 2:4 sparsity is satisfied in a tensor (i.e. every block of 4 elements has 2 zeros).”
MethodKang, C., Moore, J. A., Robertson, S., Wilms, M., Towlson, E. K., & Forkert, N. D. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107308
Using Semi-Structured Sparsity of Activations to Accelerate Neural Networks
Tatarnikova, T. M., & Raskopina, A. S. (2025) · doi:10.1109/neuront66873.2025.11049981
Adapt-S: Effective DNN Pruning via Unified Accuracy and Performance Tuning
Castro, R. L., Andrade, D., & Fraguela, B. B. (2025) · Proceedings - IEEE International Parallel and Distributed Processing Symposium · doi:10.1109/ipdps64566.2025.00018
“Examples of such techniques are Column-Vector-Wise pruning (CVW) [10], [13] which groups non-zero elements into vertical vectors of size l , or the VENOM [14] sparse format that allows to exploit arbitrary N:M sparsity patterns on Sparse Tensor Cores (SPTCs) [15], [16].”
Haris, A., Ariffin, A., Anuar, N. B., et al. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.4073
- Authors
- Mishra, A., Latorre, J. A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., & Micikevicius, P.
- Affiliations
- Nvidia (United Kingdom) (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2104.08378
- OpenAlex
- W3156528192
- Semantic Scholar
- 90d5e6f8d3b9f2617b3a3cf00fb02e730eb011cb
- Cited by
- 97
- References
- 52
- Citations by year
- 2026: 1 · 2025: 14 · 2024: 33 · 2023: 25 · 2022: 15 · 2021: 9
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
- Keywords
- sparse neural networks, model compression, inference acceleration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
