CoChat
Sign Up
PreprintCited by 4Advanced Neural Network ApplicationsTopic ModelingLow-power high-performance VLSI design

A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms

Open access
Gong, Ruihao, Ding, Yifu, Wang, Zining, Lv, Chengtao, Zheng, Xingyu, Du, Jinyang, Qin, Haotong, Guo, Jinyang +2 more·2024·arXiv
DOI 10.48550/arxiv.2409.16694arXiv 2409.16694
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) have achieved remarkable advancements in natural language processing, showcasing exceptional performance across various tasks. However, the expensive memory and computational requirements present significant challenges for their practical deployment. Low-bit quantization has emerged as a critical approach to mitigate these challenges by reducing the bit-width of model parameters, activations, and gradients, thus decreasing memory usage and computational demands. This paper presents a comprehensive survey of low-bit quantization methods tailored for LLMs, covering the fundamental principles, system implementations, and algorithmic strategies. An overview of basic concepts and new data formats specific to low-bit LLMs is first introduced, followed by a review of frameworks and systems that facilitate low-bit LLMs across various hardware platforms. Then, we categorize and analyze techniques and toolkits for efficient low-bit training and inference of LLMs. Finally, we conclude with a discussion of future trends and potential advancements of low-bit LLMs. Our systematic overview from basic, system, and algorithm perspectives can offer valuable insights and guidelines for future works to enhance the efficiency and applicability of LLMs through low-bit quantization.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2409.16694

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 148

  1. Ashkboos, S., Mohtashami, A., Croci, M.L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T., Hensman, J., 2024. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456 . · arXiv
  2. van Baalen, M., Kuzmin, A., Nagel, M., Couperus, P., Bastoul, C., Mahurin, E., Blankevoort, T., Whatmough, P., 2024. Gptvq: The blessing of dimensionality for llm quantization. arXiv preprint arXiv:2402.15319 · arXiv
  3. Bai, H., Hou, L., Shang, L., Jiang, X., King, I., Lyu, M.R., 2021a. Towards efficient post-training quantization of pre-trained language models. URL: https://arxiv.org/abs/2109.15082 , arXiv:2109.15082 . · arXiv
  4. Bai, H., Zhang, W., Hou, L., Shang, L., Jin, J., Jiang, X., Liu, Q., Lyu, M., King, I., 2021b. BinaryBERT: Pushing the limit of BERT quantization, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Association for Computational Linguistics. pp. 4334-4348.
  5. Behdin, K., Acharya, A., Gupta, A., Keerthi, S., Mazumder, R., 2023. Quantease: Optimization-based quantization for language models-an efficient and intuitive algorithm. arXiv preprint arXiv:2309.01885 . · arXiv
  6. Bondarenko, Y., Del Chiaro, R., Nagel, M., 2024. Low-rank quantizationaware training for llms. arXiv preprint arXiv:2406.06385 . · arXiv
  7. Bondarenko, Y., Nagel, M., Blankevoort, T., 2021. Understanding and overcoming the challenges of efficient transformer quantization. arXiv preprint arXiv:2109.12948 . · arXiv
  8. Cai, W.P., Li, W.J., 2024. Lcq: Low-rank codebook based quantization for large language models. arXiv preprint arXiv:2405.20973 . · arXiv
  9. Chai, Y., Gkountouras, J., Ko, G.G., Brooks, D., Wei, G.Y., 2023. Int2. 1: Towards fine-tunable quantized large language models with error correction through low-rank adaptation. arXiv preprint arXiv:2306.08162 · arXiv
  10. Chee, J., Cai, Y., Kuleshov, V., De Sa, C.M., 2024. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems 36.
  11. Chen, H., Lv, C., Ding, L., Qin, H., Zhou, X., Ding, Y., Liu, X., Zhang, M., Guo, J., Liu, X., Tao, D., 2024a. Db-llm: Accurate dualbinarization for efficient llms. URL: https://arxiv.org/abs/2402.11960 , arXiv:2402.11960 . · arXiv
  12. Chen, M., Liu, Y., Wang, J., Bin, Y., Shao, W., Luo, P., 2024b. Prefixquant: Static quantization beats dynamic through prefixed outliers in llms. arXiv preprint arXiv:2410.05265 . · arXiv
  13. Chen, M., Shao, W., Xu, P., Wang, J., Gao, P., Zhang, K., Qiao, Y., Luo, P., 2024c. Efficientqat: Efficient quantization-aware training for large language models. arXiv preprint arXiv:2407.11062 . · arXiv
  14. Cui, W., Wang, Q., 2024. Cherry on top: Parameter heterogeneity and quantization in large language models. arXiv preprint arXiv:2404.02837 · arXiv
  15. DeepSeek-AI, 2024. Deepseek-v3 technical report. URL: https://arxiv. org/abs/2412.19437 , arXiv:2412.19437 . · arXiv
  16. Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L., 2022a. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems 35, 30318-30332.
  17. Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L., 2022b. Llm.int8(): 8bit matrix multiplication for transformers at scale. URL: https://arxiv. org/abs/2208.07339 , arXiv:2208.07339 . · arXiv
  18. Dettmers, T., Lewis, M., Shleifer, S., Zettlemoyer, L., 2021. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861 . · arXiv
  19. Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L., 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems 36.
  20. Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., Alistarh, D., 2023. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078 . · arXiv
  21. Ding, Y., Feng, W., Chen, C., Guo, J., Liu, X., 2024. Reg-ptq: Regressionspecialized post-training quantization for fully quantized object detector, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16174-16184.
  22. Dong, S., Cheng, W., Qin, J., Wang, W., 2024. Qaq: Quality adaptive quantization for llm kv cache. URL: https://arxiv.org/abs/2403.04643 , arXiv:2403.04643 . · arXiv
  23. Dotzel, J., Chen, Y., Kotb, B., Prasad, S., Wu, G., Li, S., Abdelfattah, M.S., Zhang, Z., 2024a. Learning from students: Applying t-distributions to explore accurate and efficient formats for llms. arXiv preprint arXiv:2405.03103 . · arXiv
  24. Dotzel, J., Chen, Y., Kotb, B., Prasad, S., Wu, G., Li, S., Abdelfattah, M.S., Zhang, Z., 2024b. Learning from students: Applying t-distributions to explore accurate and efficient formats for llms. URL: https://arxiv. org/abs/2405.03103 , arXiv:2405.03103 . · arXiv
  25. Du, D., Zhang, Y., Cao, S., Guo, J., Cao, T., Chu, X., Xu, N., 2024. Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillation. arXiv preprint arXiv:2402.10631 . · arXiv
  26. Duanmu, H., Yuan, Z., Li, X., Duan, J., Zhang, X., Lin, D., 2024. Skvq: Sliding-window key and value cache quantization for large language models. arXiv preprint arXiv:2405.06219 . · arXiv
  27. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al., 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 . · arXiv
  28. Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., Alistarh, D., 2024a. Extreme compression of large language models via additive quantization. URL: https://arxiv.org/abs/2401.06118 , arXiv:2401.06118 . · arXiv
  29. Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., Alistarh, D., 2024b. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06118 . · arXiv
  30. Frantar, E., Alistarh, D., 2022. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems 35, 4475-4488.
  31. Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D., 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 . · arXiv
  32. Gao, C., Chen, K., Rao, J., Sun, B., Liu, R., Peng, D., Zhang, Y., Guo, X., Yang, J., Subrahmanian, V., 2024. Higher layers need more lora experts. arXiv preprint arXiv:2402.08562 . · arXiv
  33. Gong, R., Yong, Y., Gu, S., Huang, Y., Zhang, Y., Liu, X., Tao, D., 2024. Llm-qbench: A benchmark towards the best practice for post-training quantization of large language models. arXiv preprint arXiv:2405.06001 · arXiv
  34. Guo, C., Tang, J., Hu, W., Leng, J., Zhang, C., Yang, F., Liu, Y., Guo, M., Zhu, Y., 2023a. Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization, in: Proceedings of the 50th Annual International Symposium on Computer Architecture, pp. 1-15.
  35. Guo, C., Zhang, C., Leng, J., Liu, Z., Yang, F., Liu, Y., Guo, M., Zhu, Y., 2022. Ant: Exploiting adaptive numerical data type for low-bit deep neural network quantization, in: 2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO), IEEE. pp. 1414-1433.
  36. Guo, H., Greengard, P., Xing, E.P., Kim, Y., 2023b. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. arXiv preprint arXiv:2311.12023 . · arXiv
  37. Guo, J., Wu, J., Wang, Z., Liu, J., Yang, G., Ding, Y., Gong, R., Qin, H., Liu, X., 2024. Compressing large language models by joint sparsification and quantization, in: Forty-first International Conference on Machine Learning.
  38. Han, S., Mao, H., Dally, W.J., 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 . · arXiv
  39. Han, Z., Gao, C., Liu, J., Zhang, S.Q., et al., 2024. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608 . · arXiv
  40. Hassibi, B., Stork, D.G., Wolff, G.J., 1993. Optimal brain surgeon and general network pruning, in: IEEE international conference on neural networks, IEEE. pp. 293-299.
  41. Hayou, S., Ghosh, N., Yu, B., 2024. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354 . · arXiv
  42. He, Y., Zhang, L., Wu, W., Liu, J., Zhou, H., Zhuang, B., 2024. Zipcache: Accurate and efficient kv cache quantization with salient token identification. arXiv preprint arXiv:2405.14256 . · arXiv
  43. Henry, G., Tang, P.T.P., Heinecke, A., 2019. Leveraging the bfloat16 artificial intelligence datatype for higher-precision computations. URL: https://arxiv.org/abs/1904.06376 , arXiv:1904.06376 . · arXiv
  44. Heo, J.H., Kim, J., Kwon, B., Kim, B., Kwon, S.J., Lee, D., 2023. Rethinking channel dimensions to isolate outliers for low-bit weight quantization of large language models. arXiv preprint arXiv:2309.15531 . · arXiv
  45. Holmes, C., Tanaka, M., Wyatt, M., Awan, A.A., Rasley, J., Rajbhandari, S., Aminabadi, R.Y., Qin, H., Bakhtiari, A., Kurilenko, L., He, Y., 2024. Deepspeed-fastgen: High-throughput text generation for llms via mii and deepspeed-inference. URL: https://arxiv.org/abs/2401.08671 , arXiv:2401.08671 . · arXiv
  46. Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M.W., Shao, Y.S., Keutzer, K., Gholami, A., 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. arXiv preprint arXiv:2401.18079 . · arXiv
  47. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 . · arXiv
  48. Huang, W., Liao, Y., Liu, J., He, R., Tan, H., Zhang, S., Li, H., Liu, S., Qi, X., 2024a. Mc-moe: Mixture compressor for mixture-of-experts llms gains more. arXiv preprint arXiv:2410.06270 . · arXiv
  49. Huang, W., Liu, Y., Qin, H., Li, Y., Zhang, S., Liu, X., Magno, M., Qi, X., 2024b. Billm: Pushing the limit of post-training quantization for llms. arXiv preprint arXiv:2402.04291 . · arXiv
  50. Huang, X., Liu, Z., Liu, S.Y., Cheng, K.T., 2024c. Rolora: Fine-tuning rotated outlier-free llms for effective weight-activation quantization. arXiv preprint arXiv:2407.08044 . · arXiv
  51. IEEE, S.., 2019. Ieee standard for floating-point arithmetic. IEEE Std 7542019 (Revision of IEEE 754-2008) , 1-84doi: 10.1109/IEEESTD.2019. 8766229 . · DOI
  52. Jeon, H., Kim, Y., Kim, J.j., 2024. L4q: Parameter efficient quantizationaware training on large language models via lora-wise lsq. arXiv preprint arXiv:2402.04902 . · arXiv
  53. Jeong, G., Tsai, P.A., Keckler, S.W., Krishna, T., 2024. Sdq: Sparse decomposed quantization for llm inference. arXiv preprint arXiv:2406.13868 · arXiv
  54. Kang, H., Zhang, Q., Kundu, S., Jeong, G., Liu, Z., Krishna, T., Zhao, T., 2024. Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm. arXiv preprint arXiv:2403.05527 . · arXiv
  55. Kim, J., Lee, J.H., Kim, S., Park, J., Yoo, K.M., Kwon, S.J., Lee, D., 2024. Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization. Advances in Neural Information Processing Systems 36.
  56. Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M.W., Keutzer, K., 2023. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629 . · arXiv
  57. Kim, Y.J., Henry, R., Fahim, R., Awadalla, H.H., 2022. Who says elephants can't run: Bringing large scale moe models into cloud scale production. arXiv preprint arXiv:2211.10017 . · arXiv
  58. Krishnamoorthi, R., 2018. Quantizing deep convolutional networks for efficient inference: A whitepaper. URL: https://arxiv.org/abs/1806. 08342 , arXiv:1806.08342 . · arXiv
  59. Kuzmin, A., Van Baalen, M., Ren, Y., Nagel, M., Peters, J., Blankevoort, T., 2022. Fp8 quantization: The power of the exponent. Advances in Neural Information Processing Systems 35, 14651-14662.
  60. LeCun, Y., Denker, J., Solla, S., 1989. Optimal brain damage. Advances in neural information processing systems 2.
  61. Lee, C., Jin, J., Kim, T., Kim, H., Park, E., 2024a. Owq: Outlieraware weight quantization for efficient fine-tuning and inference of large language models, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 13355-13364.
  62. Lee, J., Lee, W., Sim, J., 2024b. Tender: Accelerating large language models via tensor decomposition and runtime requantization. URL: https://arxiv.org/abs/2406.12930 , arXiv:2406.12930 . · arXiv
  63. Lee, J.H., Kim, J., Kwon, S.J., Lee, D., 2023. Flexround: Learnable rounding based on element-wise division for post-training quantization, in: International Conference on Machine Learning, PMLR. pp. 1891318939.
  64. Li, B., Chen, J., Zhu, J., 2024a. Memory efficient optimizers with 4-bit states. Advances in Neural Information Processing Systems 36.
  65. Li, D., Ma, Y., Wang, N., Ye, Z., Cheng, Z., Tang, Y., Zhang, Y., Duan, L., Zuo, J., Yang, C., et al., 2024b. Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts. arXiv preprint arXiv:2404.15159 . · arXiv
  66. Li, P., Jin, X., Cheng, Y., Chen, T., 2024c. Examining post-training quantization for mixture-of-experts: A benchmark. arXiv preprint arXiv:2406.08155 . · arXiv
  67. Li, Q., Meng, R., Li, Y., Zhang, B., Li, L., Lu, Y., Chu, X., Sun, Y., Xie, Y., 2023a. A speed odyssey for deployable quantization of llms. URL: https://arxiv.org/abs/2311.09550 , arXiv:2311.09550 . · arXiv
  68. Li, Q., Zhang, Y., Li, L., Yao, P., Zhang, B., Chu, X., Sun, Y., Du, L., Xie, Y., 2023b. Fptq: Fine-grained post-training quantization for large language models. arXiv preprint arXiv:2308.15987 . · arXiv
  69. Li, S., Hu, Y., Ning, X., Liu, X., Hong, K., Jia, X., Li, X., Yan, Y., Ran, P., Dai, G., et al., 2024d. Mbq: Modality-balanced quantization for large vision-language models. arXiv preprint arXiv:2412.19509 . · arXiv
  70. Li, S., Ning, X., Hong, K., Liu, T., Wang, L., Li, X., Zhong, K., Dai, G., Yang, H., Wang, Y., 2023c. Llm-mq: Mixed-precision quantization for efficient llm deployment, in: NeurIPS 2023 Efficient Natural Language and Speech Processing Workshop, pp. 1-5.
  71. Li, S., Ning, X., Wang, L., Liu, T., Shi, X., Yan, S., Dai, G., Yang, H., Wang, Y., 2024e. Evaluating quantized large language models. arXiv preprint arXiv:2402.18158 . · arXiv
  72. Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., Chen, D., 2024f. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems 37, 22947-22970.
  73. Li, Y., Xu, S., Lin, M., Cao, X., Liu, C., Sun, X., Zhang, B., 2024g. Bi-vit: Pushing the limit of vision transformer quantization, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 3243-3251.
  74. Li, Y., Yu, Y., Liang, C., He, P., Karampatziakis, N., Chen, W., Zhao, T., 2023d. Loftq: Lora-fine-tuning-aware quantization for large language models. arXiv preprint arXiv:2310.08659 . · arXiv
  75. Lin, H., Xu, H., Wu, Y., Cui, J., Zhang, Y., Mou, L., Song, L., Sun, Z., Wei, Y., 2024a. Rotation and permutation for advanced outlier management and efficient quantization of llms. arXiv preprint arXiv:2406.01721 . · arXiv
  76. Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.M., Wang, W.C., Xiao, G., Dang, X., Gan, C., Han, S., 2024b. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems 6, 87-100.
  77. Lin, Y., Tang, H., Yang, S., Zhang, Z., Xiao, G., Gan, C., Han, S., 2024c. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. arXiv preprint arXiv:2405.04532 . · arXiv
  78. Lingle, L.D., 2023. Transformer-vq: Linear-time transformers via vector quantization. arXiv preprint arXiv:2309.16354 . · arXiv
  79. Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al., 2024a. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434 . · arXiv
  80. Liu, R., Bai, H., Lin, H., Li, Y., Gao, H., Xu, Z., Hou, L., Yao, J., Yuan, C., 2024b. Intactkv: Improving large language model quantization by keeping pivot tokens intact. arXiv preprint arXiv:2403.01241 . · arXiv
  81. Liu, S.y., Liu, Z., Huang, X., Dong, P., Cheng, K.T., 2023a. Llm-fp4: 4-bit floating-point quantized transformers. arXiv preprint arXiv:2310.16836 · arXiv
  82. Liu, Y., Li, H., Cheng, Y., Ray, S., Huang, Y., Zhang, Q., Du, K., Yao, J., Lu, S., Ananthanarayanan, G., et al., 2024c. Cachegen: Kv cache compression and streaming for fast large language model serving, in: Proceedings of the ACM SIGCOMM 2024 Conference, pp. 38-56.
  83. Liu, Y., Meng, Y., Wu, F., Peng, S., Yao, H., Guan, C., Tang, C., Ma, X., Wang, Z., Zhu, W., 2024d. Evaluating the generalization ability of quantized llms: Benchmark, analysis, and toolbox. arXiv preprint arXiv:2406.12928 . · arXiv
  84. Liu, Z., Oguz, B., Pappu, A., Shi, Y., Krishnamoorthi, R., 2023b. Binary and ternary natural language generation, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics. pp. 65-77.
  85. Liu, Z., Oguz, B., Pappu, A., Xiao, L., Yih, S., Li, M., Krishnamoorthi, R., Mehdad, Y., 2022a. Bit: Robustly binarized multi-distilled transformer, in: Advances in Neural Information Processing Systems, Curran Associates, Inc.. pp. 14303-14316.
  86. Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., Chandra, V., 2023c. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888 . · arXiv
  87. Liu, Z., Wang, Y., Han, K., Ma, S., Gao, W., 2022b. Instance-aware dynamic neural network quantization, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12434-12443.
  88. Liu, Z., Wu, B., Luo, W., Yang, X., Liu, W., Cheng, K.T., 2018. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm, in: Proceedings of the European conference on computer vision (ECCV), pp. 722-737.
  89. Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., Hu, X., 2024e. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750 . · arXiv
  90. Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Krishnamoorthi, R., Chandra, V., Tian, Y., Blankevoort, T., 2024f. Spinquant-llm quantization with learned rotations. arXiv preprint arXiv:2405.16406 · arXiv
  91. Lozhkov, A., Li, R., Allal, L.B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al., 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173 . · arXiv
  92. Luo, T., Lei, J., Lei, F., Liu, W., He, S., Zhao, J., Liu, K., 2024. Moelora: Contrastive learning guided mixture of experts on parameter-efficient fine-tuning for large language models. arXiv preprint arXiv:2402.12851 · arXiv
  93. Luo, Z., Kulmizev, A., Mao, X., 2020. Positional artefacts propagate through masked language model embeddings. arXiv preprint arXiv:2011.04393 . · arXiv
  94. Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., Wei, F., 2024a. The era of 1-bit llms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764 . · arXiv
  95. Ma, Y., Li, H., Zheng, X., Ling, F., Xiao, X., Wang, R., Wen, S., Chao, F., Ji, R., 2024b. Affinequant: Affine transformation quantization for large language models. arXiv preprint arXiv:2403.12544 . · arXiv
  96. Malinovskii, V., Mazur, D., Ilin, I., Kuznedelev, D., Burlachenko, K., Yi, K., Alistarh, D., Richtarik, P., 2024. Pv-tuning: Beyond straight-through estimation for extreme llm compression. arXiv preprint arXiv:2405.14852 · arXiv
  97. Meo, C., Sycheva, K., Goyal, A., Dauwels, J., 2024. Bayesian-lora: Lora based parameter efficient fine-tuning using optimal quantization levels and rank values trough differentiable bayesian gates. arXiv preprint arXiv:2406.13046 . · arXiv
  98. Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., et al., 2022. Fp8 formats for deep learning. arXiv preprint arXiv:2209.05433 . · arXiv
  99. Oklobdzija, V.G., 1994. An algorithmic and novel design of a leading zero detector circuit: Comparison with logic synthesis. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 2, 124-128.
  100. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., et al., 2024. Gpt-4 technical report. arXiv:2303.08774 . · arXiv
  101. Ping, B., Wang, S., Wang, H., Han, X., Xu, Y., Yan, Y., Chen, Y., Chang, B., Liu, Z., Sun, M., 2024. Delta-come: Training-free deltacompression with mixed-precision for large language models. arXiv preprint arXiv:2406.08903 . · arXiv
  102. Qin, H., Ding, Y., Zhang, M., Yan, Q., Liu, A., Dang, Q., Liu, Z., Liu, X., 2022. Bibert: Accurate fully binarized bert. arXiv preprint arXiv:2203.06390 . · arXiv
  103. Qin, H., Ma, X., Zheng, X., Li, X., Zhang, Y., Liu, S., Luo, J., Liu, X., Magno, M., 2024. Accurate lora-finetunsing quantization of llms via information retention. arXiv preprint arXiv:2402.05445 . · arXiv
  104. Rajabzadeh, H., Valipour, M., Zhu, T., Tahaei, M., Kwon, H.J., Ghodsi, A., Chen, B., Rezagholizadeh, M., 2024. Qdylora: Quantized dynamic lowrank adaptation for efficient large language model tuning. arXiv preprint
  105. Rouhani, B.D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., Dusan, S., Elango, V., Golub, M., Heinecke, A., James-Roxby, P., Jani, D., Kolhe, G., Langhammer, M., Li, A., Melnick, L., Mesmakhosroshahi, M., Rodriguez, A., Schulte, M., Shafipour, R., Shao, L., Siu, M., Dubey, P., Micikevicius, P., Naumov, M., Verrilli, C., Wittig, R., Burger, D., Chung, E., 2023. Microscaling data formats for deep learning. URL: https://arxiv.org/abs/2310.10537 , arXiv:2310.10537 . · arXiv
  106. Shang, Y., Yuan, Z., Wu, Q., Dong, Z., 2023. Pb-llm: Partially binarized large language models. arXiv preprint arXiv:2310.00034 . · arXiv
  107. Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K., Gao, P., Qiao, Y., Luo, P., 2023. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137 . · arXiv
  108. Sharify, S., Xu, Z., Wang, X., et al., 2024. Combining multiple post-training techniques to achieve most efficient quantized llms. arXiv preprint arXiv:2405.07135 . · arXiv
  109. Shen, S., Dong, Z., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M.W., Keutzer, K., 2020. Q-bert: Hessian based ultra low precision quantization of bert, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8815-8821.
  110. Shi, S., Zhao, E., Cai, D., Cui, L., Huang, X., Li, H., 2024. Inferflow: an efficient and highly configurable inference engine for large language models. URL: https://arxiv.org/abs/2401.08294 , arXiv:2401.08294 . · arXiv
  111. Sun, M., Liu, Z., Bair, A., Kolter, J.Z., 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695 . · arXiv
  112. Tambe, T., Yang, E.Y., Wan, Z., Deng, Y., Reddi, V.J., Rush, A., Brooks, D., Wei, G.Y., 2019. Adaptivfloat: A floating-point based data type for resilient deep learning inference. arXiv preprint arXiv:1909.13271 . · arXiv
  113. Tian, S., Chen, H., Lv, C., Liu, Y., Guo, J., Liu, X., Li, S., Yang, H., Xie, T., 2024. Qvd: Post-training quantization for video diffusion models, in: Proceedings of the 32nd ACM International Conference on Multimedia, pp. 10572-10581.
  114. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al., 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 . · arXiv
  115. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al., 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 . · arXiv
  116. Tseng, A., Chee, J., Sun, Q., Kuleshov, V., De Sa, C., 2024a. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396 . · arXiv
  117. Tseng, A., Sun, Q., Hou, D., De Sa, C., 2024b. Qtip: Quantization with trellises and incoherence processing. arXiv preprint arXiv:2406.11235 . · arXiv
  118. Wang, C., Wang, Z., Xu, X., Tang, Y., Zhou, J., Lu, J., 2024. Q-vlm: Posttraining quantization for large vision-language models. arXiv preprint arXiv:2410.08119 . · arXiv
  119. Wang, H., Ma, S., Dong, L., Huang, S., Wang, H., Ma, L., Yang, F., Wang, R., Wu, Y., Wei, F., 2023. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.11453 . · arXiv
  120. Warren, H., 2012. Hacker's Delight. Pearson Education. URL: https: //books.google.com.my/books?id=VicPJYM0I5QC . pages 72-73.
  121. Wei, X., Gong, R., Li, Y., Liu, X., Yu, F., 2023a. Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization. URL: https://arxiv.org/abs/2203.05740 , arXiv:2203.05740 . · arXiv
  122. Wei, X., Zhang, Y., Li, Y., Zhang, X., Gong, R., Guo, J., Liu, X., 2023b. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145 . · arXiv
  123. Wei, X., Zhang, Y., Zhang, X., Gong, R., Zhang, S., Zhang, Q., Yu, F., Liu, X., 2022. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems 35, 17402-17414.
  124. Wilkes, M.V., Wheeler, D.J., Gill, S., Corbató, F., 1958. The preparation of programs for an electronic digital computer. Reprinted 1984, pages 191-193.
  125. Wu, H., Judd, P., Zhang, X., Isaev, M., Micikevicius, P., 2020. Integer quantization for deep learning inference: Principles and empirical evaluation. URL: https://arxiv.org/abs/2004.09602 , arXiv:2004.09602 . · arXiv
  126. Xi, H., Chen, Y., Zhao, K., Zheng, K., Chen, J., Zhu, J., 2024. Jetfire: Efficient and accurate transformer pretraining with int8 data flow and per-block quantization. arXiv preprint arXiv:2403.12422 . · arXiv
  127. Xia, H., Zheng, Z., Wu, X., Chen, S., Yao, Z., Youn, S., Bakhtiari, A., Wyatt, M., Zhuang, D., Zhou, Z., et al., 2024. Fp6-llm: Efficiently serving large language models through fp6-centric algorithm-system co-design. arXiv preprint arXiv:2401.14112 . · arXiv
  128. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S., 2023. Smoothquant: Accurate and efficient post-training quantization for large language models, in: International Conference on Machine Learning, PMLR. pp. 38087-38099.
  129. Xu, Y., Xie, L., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z., Zhang, X., Tian, Q., 2023. Qa-lora: Quantization-aware low-rank adaptation of large language models. arXiv preprint arXiv:2309.14717 . · arXiv
  130. Yang, J.Y., Kim, B., Bae, J., Kwon, B., Park, G., Yang, E., Kwon, S.J., Lee, D., 2024. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. arXiv preprint arXiv:2402.18096 . · arXiv
  131. Yao, Z., Aminabadi, R.Y., Zhang, M., Wu, X., Li, C., He, Y., 2022. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. URL: https://arxiv.org/abs/2206.01861 , arXiv:2206.01861 . · arXiv
  132. Yao, Z., Wu, X., Li, C., Youn, S., He, Y., 2023. Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation. arXiv preprint arXiv:2303.08302 . · arXiv
  133. Yu, J., Zhou, S., Yang, D., Wang, S., Li, S., Hu, X., Xu, C., Xu, Z., Shu, C., Yuan, Z., 2025. Mquant: Unleashing the inference potential of multimodal large language models via full static quantization. arXiv preprint arXiv:2502.00425 . · arXiv
  134. Yuan, Z., Niu, L., Liu, J., Liu, W., Wang, X., Shang, Y., Sun, G., Wu, Q., Wu, J., Wu, B., 2023. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.01089 . · arXiv
  135. Yuan, Z., Xue, C., Chen, Y., Wu, Q., Sun, G., 2024. Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization. URL: https://arxiv.org/abs/2111.12293 , arXiv:2111.12293 . · arXiv
  136. Yue, Y., Yuan, Z., Duanmu, H., Zhou, S., Wu, J., Nie, L., 2024. Wkvquant: Quantizing weight and key/value cache for large language models gains more. arXiv preprint arXiv:2402.12065 . · arXiv
  137. Zhang, C., Cheng, J., Constantinides, G.A., Zhao, Y., 2024a. Lqer: Low-rank quantization error reconstruction for llms. arXiv preprint arXiv:2402.02446 . · arXiv
  138. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X.V., et al., 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 . · arXiv
  139. Zhang, T., Yi, J., Xu, Z., Shrivastava, A., 2024b. Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization. arXiv preprint arXiv:2405.03917 . · arXiv
  140. Zhang, W., Hou, L., Yin, Y., Shang, L., Chen, X., Jiang, X., Liu, Q., 2020. TernaryBERT: Distillation-aware ultra-low bit BERT, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics. pp. 509-521.
  141. Zhang, Y., Yang, F., Peng, S., Wang, F., Pan, A., 2024c. Flattenquant: Breaking through the inference compute-bound for large language models with per-tensor quantization. URL: https://arxiv.org/abs/2402. 17985 , arXiv:2402.17985 . · arXiv
  142. Zhang, Y., Zhang, P., Huang, M., Xiang, J., Wang, Y., Wang, C., Zhang, Y., Yu, L., Liu, C., Lin, W., 2024d. Qqq: Quality quattuor-bit quantization for large language models. arXiv preprint arXiv:2406.09904 . · arXiv
  143. Zhang, Y., Zhang, P., Huang, M., Xiang, J., Wang, Y., Wang, C., Zhang, Y., Yu, L., Liu, C., Lin, W., 2024e. Qqq: Quality quattuor-bit quantization for large language models. arXiv preprint arXiv:2406.09904 . · arXiv
  144. Zhang, Z., Jaiswal, A., Yin, L., Liu, S., Zhao, J., Tian, Y., Wang, Z., 2024f. Q-galore: Quantized galore with int4 projection and layer-adaptive lowrank gradients. arXiv preprint arXiv:2407.08296 . · arXiv
  145. Zhang, Z., Zhao, D., Miao, X., Oliaro, G., Li, Q., Jiang, Y., Jia, Z., 2024g. Quantized side tuning: Fast and memory-efficient tuning of quantized large language models. arXiv preprint arXiv:2401.07159 . · arXiv
  146. Zhao, Y., Lin, C.Y., Zhu, K., Ye, Z., Chen, L., Zheng, S., Ceze, L., Krishnamurthy, A., Chen, T., Kasikci, B., 2024. Atom: Low-bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems 6, 196-209.
  147. Zhu, F., Gong, R., Yu, F., Liu, X., Wang, Y., Li, Z., Yang, X., Yan, J., 2020. Towards unified int8 training for convolutional neural network, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1969-1979.
  148. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Braverman, V., Beidi Chen, Hu, X., 2023. Kivi : Plug-andplay 2bit kv cache quantization with streaming asymmetric quantization URL: https://rgdoi.net/10.13140/RG.2.2.28167.37282 , doi: 10. 13140/RG.2.2.28167.37282 . · DOI · Paper page

Cited by

Sorted by citations

Authors
Gong, R., Ding, Y., Wang, Z., Lv, C., Zheng, X., Du, J., Qin, H., Guo, J., Magno, M., & Liu, X.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2409.16694
arXiv
2409.16694
OpenAlex
W4403808836
Cited by
4
References
0
Citations by year
2026: 2 · 2025: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
Keywords
low-bit quantization, parameter quantization, activation quantization, gradient quantization, memory efficiency, inference efficiency, efficient training, hardware deployment, data formats

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review