CoChat
Sign Up
PreprintCited by 131Advanced Neural Network ApplicationsTopic ModelingDomain Adaptation and Few-Shot Learning

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

Open access
Frantar, Elias, Alistarh, Dan·2023·arXiv
DOI 10.48550/arxiv.2301.00774arXiv 2301.00774
Open this paper in CoChat — free accountOpen PDF

Abstract

We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy. This is achieved via a new pruning method called SparseGPT, specifically designed to work efficiently and accurately on massive GPT-family models. We can execute SparseGPT on the largest available open-source models, OPT-175B and BLOOM-176B, in under 4.5 hours, and can reach 60% unstructured sparsity with negligible increase in perplexity: remarkably, more than 100 billion weights from these models can be ignored at inference time. SparseGPT generalizes to semi-structured (2:4 and 4:8) patterns, and is compatible with weight quantization approaches. The code is available at: https://github.com/IST-DASLab/sparsegpt.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2301.00774

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • SparseGPT can help remove 100 billion parameters without any accuracy loss.

    “Techniques like SparseGPT [66] can help remove 100 billion parameters without any accuracy loss.”

    From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [66]

    The Abstract says SparseGPT can make more than 100 billion weights ignorable at inference time, but reports a negligible increase in perplexity—not no accuracy loss. The Introduction likewise describes minor accuracy loss when pruning the 175-billion-parameter OPT model. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • For a fixed target sparsity, larger models experience a much smaller accuracy drop than their smaller counterparts.

    “for a fixed target sparsity, larger models experience a much smaller accuracy drop than their smaller counterparts.”

    From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [66]

  • The most extensive models from the OPT and BLOOM families can be pruned to 50% sparsity with a minimal increase in perplexity.

    “the most extensive models from the OPT and BLOOM families can be pruned to 50% sparsity with a minimal increase in perplexity [66].”

    From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [66]

  • SparseGPT can be extended to N:M sparsity (2:4 and 4:8) with some additional accuracy loss compared to the unoptimized model.

    “SparseGPT can be further extended to N:M sparsity (2:4 and 4:8) with some additional accuracy loss compared to the unoptimized model.”

    From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [66]

  • Frantar et al. show that it is possible to remove 50% of the weight parameters without significantly compromising model perplexity.

    “Frantar et al. [66] propose a static PTP method to compress giant language models, such as GPT, and show that it is possible to remove 50% of the weight parameters without significantly compromising model perplexity.”

    From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [66]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 131

Research works printed but not linked: 48

  1. Blumensath, T. and Davies, M. E. Iterative thresholding for sparse approximations. Journal of Fourier Analysis and Applications, 14(5-6):629-654, 2008.
  2. Boratko, M., Padigela, H., Mikkilineni, D., Yuvraj, P., Das, R., McCallum, A., Chang, M., Fokoue-Nkoutche, A., Ka- panipathi, P., Mattei, N., et al. A systematic classification of knowledge, reasoning, and context within the ARC dataset. arXiv preprint arXiv:1806.00358, 2018. · arXiv
  3. Dettmers, T. and Zettlemoyer, L. The case for 4-bit pre- cision: k-bit inference scaling laws. arXiv preprint arXiv:2212.09720, 2022. · arXiv
  4. Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. LLM.int8(): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022. · arXiv
  5. Elsen, E., Dukhan, M., Gale, T., and Simonyan, K. Fast sparse convnets. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  6. Evci, U., Gale, T., Menick, J., Castro, P. S., and Elsen, E. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning (ICML), 2020.
  7. Frantar, E. and Alistarh, D. SPDY: Accurate pruning with speedup guarantees. arXiv preprint arXiv:2201.13096, 2022. · arXiv
  8. Frantar, E., Kurtic, E., and Alistarh, D. M-FAC: Efficient matrix-free approximations of second-order information. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
  9. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210.17323, 2022a. · arXiv
  10. Frantar, E., Singh, S. P., and Alistarh, D. Optimal Brain Compression: A framework for accurate post- training quantization and pruning. arXiv preprint arXiv:2208.11580, 2022b. Accepted to NeurIPS 2022, to appear. · arXiv
  11. Gale, T., Elsen, E., and Hooker, S. The state of sparsity in deep neural networks. In International Conference on Machine Learning (ICML), 2019.
  12. Hagiwara, M. A simple and effective method for removal of hidden units and weights. Neurocomputing, 6(2):207 - 218, 1994. ISSN 0925-2312.
  13. Han, S., Pool, J., Tran, J., and Dally, W. J. Learning both weights and connections for efficient neural networks. In Conference on Neural Information Processing Systems (NeurIPS), 2015.
  14. Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. In International Con- ference on Learning Representations (ICLR), 2016.
  15. Hassibi, B., Stork, D. G., and Wolff, G. J. Optimal brain sur- geon and general network pruning. In IEEE International Conference on Neural Networks, 1993.
  16. He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., and Han, S. AMC: AutoML for model compression and acceleration on mobile devices. In European Conference on Computer Vision (ECCV), 2018.
  17. Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., and Peste, A. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. arXiv preprint arXiv:2102.00554, 2021. · arXiv
  18. Hubara, I., Chmiel, B., Island, M., Banner, R., Naor, S., and Soudry, D. Accelerated sparse neural training: A provable and efficient method to find N:M transposable masks. In Conference on Neural Information Processing Systems (NeurIPS), 2021a.
  19. Hubara, I., Nahshan, Y., Hanani, Y., Banner, R., and Soudry, D. Accurate post training quantization with small cal- ibration sets. In International Conference on Machine Learning (ICML), 2021b.
  20. Kingdon, J. Hypothesising Neural Nets, pp. 81-106. Springer London, London, 1997. ISBN 978-1-4471-0949- 5. doi: 10.1007/978-1-4471-0949-5 5. · DOI
  21. Kurtic, E. and Alistarh, D. Gmp*: Well-tuned global magni- tude pruning can outperform most bert-pruning methods. arXiv preprint arXiv:2210.06384, 2022. · arXiv
  22. Kurtic, E., Campos, D., Nguyen, T., Frantar, E., Kurtz, M., Fineran, B., Goin, M., and Alistarh, D. The Op- timal BERT Surgeon: Scalable and accurate second- order pruning for large language models. arXiv preprint arXiv:2203.07259, 2022. · arXiv
  23. Kurtz, M., Kopinsky, J., Gelashvili, R., Matveev, A., Carr, J., Goin, M., Leiserson, W., Moore, S., Nell, B., Shavit, N., and Alistarh, D. Inducing and exploiting activation sparsity for fast inference on deep neural networks. In International Conference on Machine Learning (ICML), 2020.
  24. Kwon, W., Kim, S., Mahoney, M. W., Hassoun, J., Keutzer, K., and Gholami, A. A fast post-training pruning frame- work for transformers. arXiv preprint arXiv:2204.09656, 2022. · arXiv
  25. LeCun, Y., Denker, J. S., and Solla, S. A. Optimal brain damage. In Conference on Neural Information Processing Systems (NeurIPS), 1989.
  26. Li, Y., Gong, R., Tan, X., Yang, Y., Hu, P., Zhang, Q., Yu, F., Wang, W., and Gu, S. BRECQ: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations (ICLR), 2021.
  27. Liu, L., Zhang, S., Kuang, Z., Zhou, A., Xue, J.-H., Wang, X., Chen, Y., Yang, W., Liao, Q., and Zhang, W. Group fisher pruning for practical network compression. In International Conference on Machine Learning (ICML), 2021.
  28. Marcus, M., Kim, G., Marcinkiewicz, M. A., MacIntyre, R., Bies, A., Ferguson, M., Katz, K., and Schasberger, B. The penn treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994.
  29. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. · arXiv
  30. Mishra, A., Latorre, J. A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P. Ac- celerating sparse deep neural networks. arXiv preprint arXiv:2104.08378, 2021. · arXiv
  31. Mostafazadeh, N., Roth, M., Louis, A., Chambers, N., and Allen, J. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pp. 46-51, 2017.
  32. Nagel, M., Amjad, R. A., Van Baalen, M., Louizos, C., and Blankevoort, T. Up or down? Adaptive rounding for post-training quantization. In International Conference on Machine Learning (ICML), 2020.
  33. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word predic- tion requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016. · arXiv
  34. Park, G., Park, B., Kwon, S. J., Kim, B., Lee, Y., and Lee, D. nuQmm: Quantized matmul for efficient inference of large-scale generative language models. arXiv preprint arXiv:2206.09557, 2022a. · arXiv
  35. Park, M., You, J., Nagel, M., and Chang, S. Quadapter: Adapter for gpt-2 quantization. arXiv preprint arXiv:2211.16912, 2022b. · arXiv
  36. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In Conference on Neural Informa- tion Processing Systems (NeurIPS), 2019.
  37. Peste, A., Iofinova, E., Vladu, A., and Alistarh, D. AC/DC: Alternating compressed/decompressed training of deep neural networks. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
  38. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 (140):1-67, 2020.
  39. Sanh, V., Wolf, T., and Rush, A. M. Movement prun- ing: Adaptive sparsity by fine-tuning. arXiv preprint arXiv:2005.07683, 2020. · arXiv
  40. Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. · arXiv
  41. Singh, S. P. and Alistarh, D. WoodFisher: Efficient second- order approximation for neural network compression. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
  42. Tata, S. and Patel, J. M. PiQA: An algebra for querying pro- tein data sets. In International Conference on Scientific and Statistical Database Management, 2003.
  43. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. · arXiv
  44. Xiao, G., Lin, J., Seznec, M., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438, 2022. · arXiv
  45. Yao, Z., Aminabadi, R. Y., Zhang, M., Wu, X., Li, C., and He, Y. ZeroQuant: Efficient and affordable post-training quantization for large-scale transformers. arXiv preprint arXiv:2206.01861, 2022. · arXiv
  46. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. · arXiv
  47. Zhou, A., Ma, Y., Zhu, J., Liu, J., Zhang, Z., Yuan, K., Sun, W., and Li, H. Learning N:M fine-grained structured sparse neural networks from scratch. In International Conference on Learning Representations (ICLR), 2021.
  48. Zhu, M. and Gupta, S. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017. · arXiv

Authors
Frantar, E., & Alistarh, D.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2301.00774
arXiv
2301.00774
OpenAlex
W4313484599
Cited by
131
References
0
Citations by year
2025: 49 · 2024: 63 · 2023: 19
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Institute of Science and Technology Austria
  • European Commission (805223, HORIZON2020)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
neural network pruning, one-shot pruning, large language models, unstructured sparsity, weight quantization, inference efficiency, perplexity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review