CoChat
Sign Up
PreprintCited by 34Advanced Neural Network ApplicationsTopic ModelingDomain Adaptation and Few-Shot Learning

BiLLM: Pushing the Limit of Post-Training Quantization for LLMs

Open access
Huang, Wei, Liu, Yangdong, Qin, Haotong, Li, Ying, Zhang, Shiming, Liu, Xianglong, Magno, Michele, Qi, Xiaojuan·2024·arXiv
DOI 10.48550/arxiv.2402.04291arXiv 2402.04291
Open this paper in CoChat — free accountOpen PDF

Abstract

Pretrained large language models (LLMs) exhibit exceptional general language processing capabilities but come with significant demands on memory and computational resources. As a powerful compression technology, binarization can extremely reduce model weights to a mere 1 bit, lowering the expensive computation and memory requirements. However, existing quantization techniques fall short of maintaining LLM performance under ultra-low bit-widths. In response to this challenge, we present BiLLM, a groundbreaking 1-bit post-training quantization scheme tailored for pretrained LLMs. Based on the weight distribution of LLMs, BiLLM first identifies and structurally selects salient weights, and minimizes the compression loss through an effective binary residual approximation strategy. Moreover, considering the bell-shaped distribution of the non-salient weights, we propose an optimal splitting search to group and binarize them accurately. BiLLM achieving for the first time high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit weights across various LLMs families and evaluation metrics, outperforms SOTA quantization methods of LLM by significant margins. Moreover, BiLLM enables the binarization process of the LLM with 7 billion weights within 0.5 hours on a single GPU, demonstrating satisfactory time efficiency. Our code is available at https://github.com/Aaronhuang-778/BiLLM.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.04291

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Even 1-bit models (e.g., approximately 15.4 GB for 70B parameters) may exceed the capacity of highly resource-constrained devices.

    “Nevertheless, even 1-bit models ( e . g ., approximately 15.4 GB for 70B parameters [7]) may exceed the capacity of highly resource-constrained devices [15].”

    From LittleBit: Ultra Low-Bit Quantization via Latent Factorization (2025) · cites this paper as [7]

  • LITTLEBIT remains competitive with PTQ methods such as BiLLM.

    “It also remains competitive with PTQ methods such as BiLLM [7].”

    From LittleBit: Ultra Low-Bit Quantization via Latent Factorization (2025) · cites this paper as [7]

  • PTQ methods generally struggle to maintain performance below 2-bit precision.

    “However, these methods generally struggle to maintain performance below 2-bit precision [33, 7].”

    From LittleBit: Ultra Low-Bit Quantization via Latent Factorization (2025) · cites this paper as [7]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 55

  1. Bengio, Y., Léonard, N., and Courville, A. Estimating or prop- agating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013. · arXiv
  2. Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
  3. Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural network. In International confer- ence on machine learning, pp. 1613-1622. PMLR, 2015.
  4. Chan, C.-Y. and Ioannidis, Y. E. Bitmap index design and evalua- tion. In Proceedings of the 1998 ACM SIGMOD international conference on Management of data, pp. 355-366, 1998.
  5. Chee, J., Cai, Y., Kuleshov, V., and De Sa, C. M. Quip: 2-bit quan- tization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
  6. Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019. · arXiv
  7. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. · arXiv
  8. Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., and Bengio, Y. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830, 2016. · arXiv
  9. Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Llm. int8 (): 8-bit matrix multiplication for transformers at scale, 2022. CoRR abs/2208.07339. · arXiv
  10. Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022. · arXiv
  11. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023a. · arXiv
  12. Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078, 2023b. · arXiv
  13. Dong, Z., Yao, Z., Gholami, A., Mahoney, M. W., and Keutzer, K. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 293-302, 2019.
  14. Fang, J., Shafiee, A., Abdel-Aziz, H., Thorsley, D., Georgiadis, G., and Hassoun, J. H. Post-training piecewise linear quantization for deep neural networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 69-86. Springer, 2020.
  15. Faraone, J., Fraser, N., Blott, M., and Leong, P. H. Syq: Learning symmetric quantization for efficient deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4300-4309, 2018.
  16. Frantar, E. and Alistarh, D. Optimal brain compression: A frame- work for accurate post-training quantization and pruning. Ad- vances in Neural Information Processing Systems, 35:4475- 4488, 2022.
  17. Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Confer- ence on Machine Learning, pp. 10323-10337. PMLR, 2023.
  18. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022. · arXiv
  19. Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W., and Keutzer, K. Ai and memory wall. IEEE Micro, 2024.
  20. Helwegen, K., Widdicombe, J., Geiger, L., Liu, Z., Cheng, K.-T., and Nusselder, R. Latent weights do not exist: Rethinking binarized neural network optimization. Advances in neural information processing systems, 32, 2019.
  21. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018.
  22. Jain, S., Venkataramani, S., Srinivasan, V., Choi, J., Gopalakrish- nan, K., and Chang, L. Biscaled-dnn: Quantizing long-tailed datastructures with two scale factors for deep neural networks. In Proceedings of the 56th Annual Design Automation Confer- ence 2019, pp. 1-6, 2019.
  23. LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. Ad- vances in neural information processing systems, 2, 1989.
  24. Lee, C., Jin, J., Kim, T., Kim, H., and Park, E. Owq: Lessons learned from activation outliers for weight quantization in large language models. arXiv preprint arXiv:2306.02272, 2023. · arXiv
  25. Li, Y., Gong, R., Tan, X., Yang, Y., Hu, P., Zhang, Q., Yu, F., Wang, W., and Gu, S. Brecq: Pushing the limit of post- training quantization by block reconstruction. arXiv preprint arXiv:2102.05426, 2021. · arXiv
  26. Li, Z., Ni, B., Zhang, W., Yang, X., and Gao, W. Performance guaranteed network acceleration via high-order residual quanti- zation. In Proceedings of the IEEE international conference on computer vision, pp. 2584-2592, 2017.
  27. Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023. · arXiv
  28. Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888, 2023. · arXiv
  29. Marcus, M., Kim, G., Marcinkiewicz, M. A., MacIntyre, R., Bies, A., Ferguson, M., Katz, K., and Schasberger, B. The penn treebank: Annotating predicate argument structure. In Hu- man Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994.
  30. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. · arXiv
  31. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018. · arXiv
  32. Park, E., Yoo, S., and Vajda, P. Value-aware quantization for training and inference of neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 580- 595, 2018.
  33. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Py- torch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  34. Qin, H., Gong, R., Liu, X., Shen, M., Wei, Z., Yu, F., and Song, J. Forward and backward information retention for accurate binary neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2250-2259, 2020.
  35. Qin, H., Ding, Y., Zhang, M., Yan, Q., Liu, A., Dang, Q., Liu, Z., and Liu, X. Bibert: Accurate fully binarized bert. arXiv preprint arXiv:2203.06390, 2022. · arXiv
  36. Qin, H., Zhang, M., Ding, Y., Li, A., Cai, Z., Liu, Z., Yu, F., and Liu, X. Bibench: Benchmarking and analyzing network binarization. arXiv preprint arXiv:2301.11233, 2023. · arXiv
  37. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of trans- fer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5551, 2020.
  38. Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. Xnor- net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-542. Springer, 2016.
  39. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Wino- grande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
  40. Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. Multitask prompted training enables zero-shot task gener- alization. arXiv preprint arXiv:2110.08207, 2021. · arXiv
  41. Shang, Y., Yuan, Z., Wu, Q., and Dong, Z. Pb-llm: Partially bina- rized large language models. arXiv preprint arXiv:2310.00034, 2023. · arXiv
  42. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. · arXiv
  43. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. · arXiv
  44. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  45. Wang, H., Ma, S., Dong, L., Huang, S., Wang, H., Ma, L., Yang, F., Wang, R., Wu, Y., and Wei, F. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.11453, 2023. · arXiv
  46. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. · arXiv
  47. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Hug- gingface's transformers: State-of-the-art natural language pro- cessing. arXiv preprint arXiv:1910.03771, 2019. · arXiv
  48. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023.
  49. Yao, Z., Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y. Z. Efficient and affordable post-training quantiza- tion for large-scale transformers, 2022. URL https://arxiv. org/abs/2206.01861.
  50. Yao, Z., Li, C., Wu, X., Youn, S., and He, Y. A comprehensive study on post-training quantization for large language models. arXiv preprint arXiv:2303.08302, 2023. · arXiv
  51. You, Y. Audio coding: theory and applications. Springer Science & Business Media, 2010.
  52. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. · arXiv
  53. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. · arXiv
  54. Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., and Zou, Y. Dorefa- net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160, 2016. · arXiv
  55. Zhu, X., Li, J., Liu, Y., Ma, C., and Wang, W. A survey on model compression for large language models. arXiv preprint arXiv:2308.07633, 2023. 83.23 15.38 141.09 35.85 47.54 12.26 ptb c4 LLaMA-30B GPTQ-2bits PB-LLM-1.7bits BiLLM-1.1bits 20.86 15.13 35.05 25.29 21.24 16.15 ptb c4 OPT-30B GPTQ-2bits PB-LLM-1.7bits BiLLM-1.1bits 163.72 69.07 278.52 180.05 103.82 66.51 ptb c4 OPT-1.3B GPTQ-2bits PB-LLM-1.7bits BiLLM-1.1bits 87.22 57.75 143.93 96.13 76.99 45.82 ptb c4 OPT-2.7B GPTQ-2bits PB-LLM-1.7bits BiLLM-1.1bits 63.9 11.56 66.8 16.92 46.99 11.12 ptb c4 LLaMA-65B GPTQ-2bits PB-LLM-1.7bits BiLLM-1.1bits 109.49 61.01 45.24 34.98 18.57 14.13 ptb c4 OPT-66B GPTQ-2bits PB-LLM-1.7bits BiLLM-1.1bits 470.34 29.47 719.25 144.59 332.09 27.54 ptb c4 LLaMA-2-13B GPTQ-2bits PB-LLM-1.7bits BiLLM-1.1bits 27.16 18.83 110.47 53.26 26.98 19.89 ptb c4 OPT-13B GPTQ-2bits PB-LLM-1.7bits BiLLM-1.1bits 196.7 21.2 213.17 39.51 85.32 16.93 ptb c4 LLaMA-13B GPTQ-2bits PB-LLM-1.7bits BiLLM-1. · arXiv

Cited by

Sorted by citations

Related

Authors
Huang, W., Liu, Y., Qin, H., Li, Y., Zhang, S., Liu, X., Magno, M., & Qi, X.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.04291
arXiv
2402.04291
OpenAlex
W4391670689
Cited by
33
References
0
Citations by year
2025: 15 · 2024: 18
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation
  • Baidu
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (219943)
  • National Natural Science Foundation of China (62306025, 92367204)
  • National Science and Technology Major Project
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
post-training quantization, large language models, weight distribution, model compression

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review