The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration
Abstract
Large language models (LLMs) have transformed numerous AI applications. On-device LLM is becoming increasingly important: running LLMs locally on edge devices can reduce cloud computing costs and protect users' privacy. However, the astronomical model size and the limited hardware resources pose significant deployment challenges. To solve these issues, we propose Activation-aware Weight Quantization (AWQ) and TinyChat, an algorithm-system full-stack solution for efficient on-device LLM deployment. AWQ is a novel quantization method that identifies and protects salient weights based on activation distribution, significantly reducing model size while preserving performance. TinyChat, an optimized inference framework, translates AWQ's theoretical memory savings into practical speedups through techniques such as on-the-fly dequantization, SIMD-aware weight packing, and kernel fusion. Together, they enable 4x model size reduction and 3-4x acceleration across various edge platforms, from high-end desktop GPUs to resource-constrained IoT devices. This solution democratizes on-device LLM deployment, offering privacy-preserving, low-latency AI capabilities across a wide range of applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“To further reduce the quantization error, recent studies propose to protect important weights that are more sensitive to quantization, known as salient weights [305-306].”
From A Survey of Large Language Models (2026) · cites this paper as [305]
In the citing paper
“While post-training quantization (PTQ) methods such as GPTQ [8] and AWQ[9] effectively compress models to approximately 4 bits”
From LittleBit: Ultra Low-Bit Quantization via Latent Factorization (2025) · cites this paper as [9]
In the citing paper
“More broadly, there is a rich literature on optimizing inference speed, both for diffusion models [52, 38, 56, 17] and large transformers in general [32, 25, 35].”
From Real-Time Execution of Action Chunking Flow Policies (2025) · cites this paper as [35]
Citations
Cites
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
Wang, H., Zhang, Z., & Han, S. (2021) · 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca51647.2021.00018 · Cited by 427
Self-citationLearned Step Size Quantization
Esser, S. K., McKinstry, J. L., Bablani, D., Appuswamy, R., & Modha, D. S. (2019) · arXiv · doi:10.48550/arxiv.1902.08153 · Cited by 415
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2208.07339 · Cited by 305
LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale
Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2198 · Cited by 305
Wei, X., Zhang, Y., Li, Y., Zhang, X., Gong, R., Guo, J., & Liu, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.102 · Cited by 258
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., & Han, S. (2022) · arXiv · doi:10.48550/arxiv.2211.10438 · Cited by 247
Self-citationVILA: On Pre-training for Visual Language Models
Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., & Han, S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02520 · Cited by 243
Self-citationGPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022) · arXiv · doi:10.48550/arxiv.2210.17323 · Cited by 197
The case for 4-bit precision: k-bit Inference Scaling Laws
Dettmers, T., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2212.09720 · Cited by 46
Cited by
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
NVILA: Efficient Frontier Visual Language Models
Liu, Z., Zhu, L., Shi, B., Zhang, Z., Lou, Y., Yang, S. W., Xi, H., Cao, S., Gu, Y., Li, D., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00390 · Cited by 334
Self-citationEdgeMoE: Empowering Sparse Large Language Models on Mobile Devices
Yi, R., Guo, L., Wei, S., Zhou, A., Wang, S., & Xu, M. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3546466 · Cited by 224
Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Jin, H., Chen, T., & Jia, Z. (2025) · ACM Computing Surveys · doi:10.1145/3754448 · Cited by 234
A Review on Edge Large Language Models: Design, Execution, and Applications
Zheng, Y., Chen, Y., Qian, B., Shi, X., Shu, Y., & Chen, J. (2025) · ACM Computing Surveys · doi:10.1145/3719664 · Cited by 169
Metadata
- Authors
- Lin, J., Tang, J., Tang, H., Yang, S., Xiao, G., & Han, S.
- Affiliations
- Massachusetts Institute of Technology (US)
- Venue
- GetMobile: Mobile Computing and Communications
- Published
- 2025-01-20
- Volume
- 28
- Issue
- 4
- Pages
- 12-17
- ISSN
- 2375-0529 (ISSN-L)
- 2375-0537
- OpenAlex
- W4406650295
- Cited by
- 274
- References
- 10
- Citations by year
- 2026: 106 · 2025: 157
- FWCI
- 92.34
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- GetMobile Mobile Computing and Communications (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- on-device large language models, model compression, edge deployment, kernel fusion, activation distribution, privacy-preserving inference, low-latency inference, IoT devices
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. https://app.cochat.ai/papers/10.1145/3714983.3714987
- The paper
- Lin, J., Tang, J., Tang, H., Yang, S., Xiao, G., & Han, S. (2025). AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. GetMobile: Mobile Computing and Communications. https://doi.org/10.1145/3714983.3714987
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.