The DOI matches its Crossref record; 104 of the 114 references it cites were found in a registry or an index.
A Survey on Model Compression for Large Language Models
Open accessAbstract
Large Language Models (LLMs) have transformed natural language processing tasks successfully. Yet, their large size and high computational needs pose challenges for practical use, especially in resource-limited settings. Model compression has emerged as a key research area to address these challenges. This paper presents a survey of model compression techniques for LLMs. We cover methods like quantization, pruning, and knowledge distillation, highlighting recent advancements. We also discuss benchmarking strategies and evaluation metrics crucial for assessing compressed LLMs. This survey offers valuable insights for researchers and practitioners, aiming to enhance efficiency and real-world applicability of LLMs while laying a foundation for future advancements.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Active efforts in the machine learning community to shrink and streamline LLMs may also help improve their efficiency.43”
From Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions (2026) · cites this paper as [43]
In the citing paper
“TLMs are typically created by employing techniques such as knowledge distillation, pruning, quantization [17,18,19,20,21,22,23], and specialized architectures [24] designed to minimize model size and memory footprint.”
From Tiny Language Models for Automation and Control: Overview, Potential Applications, and Future Research Directions (2025) · cites this paper as [18]
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,869
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Neural Architecture Search with Reinforcement LearningPreprint
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 5,705
Pointer Sentinel Mixture ModelsPreprint
Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07843 · Cited by 5,058
Flagged first, then by citations
10 of the 114 references couldn’t be matched to a known paper.
Research works printed but not linked: 13
- Agarwal, 2024, Generalized knowledge distillation for auto-regressive language models
- Chee, 2023, QuIP: 2-bit quantization of large language models with guarantees
- Dettmers, 2022, Gpt3.int8(): 8-bit matrix multiplication for transformers at scale
- Frantar, 2023, OPTQ: Accurate quantization for generative pre-trained transformers
- Yuxian, 2024, MiniLLM: Knowledge distillation of large language models
- Han, 2016, Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding
- Han, 2015, Learning both weights and connections for efficient neural network
- Kim, 2023, Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization
- Liu, 2024, Learning to reason with autoregressive in-context distillation
- Liu, 2024, KIVI: A tuning-free asymmetric 2bit quantization for KV cache, CoRR
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
- Sharma, 2024, The truth is in there: Improving reasoning with layer-selective rank reduction
- Williams, 2023, How does calibration data affect the post-training pruning and quantization of large language models?, CoRR
Cited by
Sorted by citations
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations
Cheng, H., Zhang, M., & Shi, Q. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3447085 · Cited by 667
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Resource-efficient Algorithms and Systems of Foundation Models: A Survey
Xu, M., Cai, D., Yin, W., Wang, S., Jin, X., & Liu, X. (2024) · ACM Computing Surveys · doi:10.1145/3706418 · Cited by 255
Metadata
- Authors
- Zhu, X., Li, J., Liu, Y., Ma, C., & Wang, W.
- Affiliations
- Chinese Academy of Sciences (CN)
- Beijing Normal University (CN)
- Renmin University of China (CN)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 12
- Pages
- 1556-1577
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4404918643
- Semantic Scholar
- 338d8f3b199abcebc85f34016b0162ab3a9d5310
- Cited by
- 273
- References
- 114
- Citations by year
- 2026: 99 · 2025: 123 · 2024: 45
- FWCI
- 37.71
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62076234, 62106257)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- model compression, large language models, quantization, pruning, knowledge distillation, resource-constrained deployment, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
