CoChat

AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration

Lin, Ji, Tang, Jiaming, Tang, Haotian, Yang, Shang, Xiao, Guangxuan, Han, Song · 2025 · GetMobile: Mobile Computing and Communications

The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.

Abstract

Large language models (LLMs) have transformed numerous AI applications. On-device LLM is becoming increasingly important: running LLMs locally on edge devices can reduce cloud computing costs and protect users' privacy. However, the astronomical model size and the limited hardware resources pose significant deployment challenges. To solve these issues, we propose Activation-aware Weight Quantization (AWQ) and TinyChat, an algorithm-system full-stack solution for efficient on-device LLM deployment. AWQ is a novel quantization method that identifies and protects salient weights based on activation distribution, significantly reducing model size while preserving performance. TinyChat, an optimized inference framework, translates AWQ's theoretical memory savings into practical speedups through techniques such as on-the-fly dequantization, SIMD-aware weight packing, and kernel fusion. Together, they enable 4x model size reduction and 3-4x acceleration across various edge platforms, from high-end desktop GPUs to resource-constrained IoT devices. This solution democratizes on-device LLM deployment, offering privacy-preserving, low-latency AI capabilities across a wide range of applications.

Overview

Integrity signals

Registry

Crossref · 10.1145/3714983.3714987

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Cited by

See all 332

Metadata

Authors
Lin, J., Tang, J., Tang, H., Yang, S., Xiao, G., & Han, S.
Affiliations
Massachusetts Institute of Technology (US)
Venue
GetMobile: Mobile Computing and Communications
Published
2025-01-20
Volume
28
Issue
4
Pages
12-17
ISSN
  • 2375-0529 (ISSN-L)
  • 2375-0537
OpenAlex
W4406650295
Cited by
274
References
10
Citations by year
2026: 106 · 2025: 157
FWCI
92.34
Type
journal article
Language
en
Open access
closed
Locations
GetMobile Mobile Computing and Communications (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
Keywords
on-device large language models, model compression, edge deployment, kernel fusion, activation distribution, privacy-preserving inference, low-latency inference, IoT devices

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.