CoChat
Sign Up
PreprintCited by 703Advanced Neural Network ApplicationsLow-power high-performance VLSI designNumerical Methods and Algorithms

PACT: Parameterized Clipping Activation for Quantized Neural Networks

Open access
Choi, Jungwook, Wang, Zhuo, Venkataramani, Swagath, Chuang, Pierce I-Jen, Srinivasan, Vijayalakshmi, Gopalakrishnan, Kailash·2018·arXiv
DOI 10.48550/arxiv.1805.06085arXiv 1805.06085
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep learning algorithms achieve high classification accuracy at the expense of significant computation cost. To address this cost, a number of quantization schemes have been proposed - but most of these techniques focused on quantizing weights, which are relatively smaller in size compared to activations. This paper proposes a novel quantization scheme for activations during training - that enables neural networks to work well with ultra low precision weights and activations without any significant accuracy degradation. This technique, PArameterized Clipping acTivation (PACT), uses an activation clipping parameter $α$ that is optimized during training to find the right quantization scale. PACT allows quantizing activations to arbitrary bit precisions, while achieving much better accuracy relative to published state-of-the-art quantization schemes. We show, for the first time, that both weights and activations can be quantized to 4-bits of precision while still achieving accuracy comparable to full precision networks across a range of popular models and datasets. We also show that exploiting these reduced-precision computational units in hardware can enable a super-linear improvement in inferencing performance due to a significant reduction in the area of accelerator compute engines coupled with the ability to retain the quantized model and activation data in on-chip memories.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1805.06085

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The first and last layers of convolutional neural networks are invariably implemented with high precision FP, even in studies claiming CNN iso-accuracy at very low fixed-point precision.

    “For instance, the first and last layers of convolutional neural networks (CNNs) are invariably implemented with high precision FP, even in studies claiming CNN iso-accuracy at very low fixed-point precision7,8.”

    From Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators (2023) · cites this paper as [8]

    Appendix A.3 (p. 11) says many previous studies kept the first and last layers at full precision, but does not call the practice invariable. It then reports little accuracy degradation when those layers are quantized to 8 bits (p. 12), directly contradicting the claim that they are invariably implemented in high-precision floating point. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • The accuracy gap mainly comes from low-precision (≤4-bit) quantization of inputs and activations, especially on the most sensitive input and output layers.

    “The accuracy gap mainly comes from low-precision (≤4-bit) quantization of inputs and activations, especially on the most sensitive input and output layers46.”

    From A compute-in-memory chip based on resistive random-access memory (2022) · cites this paper as [46]

    Section A.3 (pp. 11–12) finds that aggressively quantizing the first or last layer can noticeably degrade accuracy, while using 8 bits for those layers causes little degradation in the tested model. Section 5.1 (p. 7) also finds that activation accuracy improves with precision. But it does not establish that the accuracy gap mainly comes from inputs and activations, and it reports less than a 1% validation-error difference when activation precision is at least 4 bits, so the claim’s ≤4-bit framing is too broad. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Dynamic quantization for activations generally results in improved accuracy compared to static quantization, though with higher computational overhead.

    “Although this method introduces higher computational overhead, it generally results in improved accuracy [26].”

    From HADQ-Net: A Power-Efficient and Hardware-Adaptive Deep Convolutional Neural Network Translator Based on Quantization-Aware Training for Hardware Accelerators (2025) · cites this paper as [26]

    Section 5.1 (p. 7) reports better accuracy for PACT than for the activation-quantization schemes it tested. But the paper does not establish a general dynamic-versus-static quantization comparison or say that the more accurate approach incurs higher computational overhead; its Introduction (pp. 1–2) describes scales learned during training instead. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Quantization compresses a neural network by using lower bit-width for weight values without changing the model architecture, which is particularly useful for carefully designed network architectures like transformers.

    “Among various compression methods like pruning [16, 29, 19] and weight decomposition [36], quantization method [9, 37, 7, 27, 14] compresses a neural network by using lower bit-width for weight values without changing the model architecture, which is particularly useful for carefullydesigned network architectures like transformers.”

    From Post-Training Quantization for Vision Transformer (2021) · cites this paper as [7]

    The Introduction (p. 1) says that reducing the bit-precision of weights and activations can lower storage requirements, supporting the compression claim. The paper discusses CNNs, not transformers, and does not establish that quantization is particularly useful for carefully designed transformer architectures. It also does not explicitly state that the model architecture remains unchanged. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Sensitive input and output layers have been reported to cause large accuracy degradation and are thus often excluded from low-precision quantization.

    “which have been reported to cause large accuracy degradation and are thus often excluded from low-precision quantization46,54.”

    From A compute-in-memory chip based on resistive random-access memory (2022) · cites this paper as [46]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 24

  1. Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. CoRR , abs/1308.3432, 2013. · arXiv
  2. Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep Learning With Low Precision by Half-Wave Gaussian Quantization. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , July 2017.
  3. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. BinaryConnect: Training Deep Neural Networks with binary weights during propagations. CoRR , abs/1511.00363, 2015. · arXiv
  4. Benjamin Graham. Low-Precision Batch-Normalized Activations. CoRR , abs/1702.08231, 2017. · arXiv
  5. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity Mappings in Deep Residual Networks. CoRR , abs/1603.05027, 2016a. · arXiv
  6. Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized Neural Networks. NIPs , pp. 4107-4115, 2016a.
  7. Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations. CoRR , abs/1609.07061, 2016b. · arXiv
  8. Kyuyeon Hwang and Wonyong Sung. Fixed-point Feedforward Deep Neural Network Design Using Weights +1, 0, and -1. In IEEE Workshop on Signal Processing Systems (SiPS) , pp. 1-6, Oct. 2014.
  9. Minje Kim and Paris Smaragdis. Bitwise Neural Networks. ICML Workshop on Resource-Efficient Machine Learning , 2015.
  10. Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (ICLR) , 2015.
  11. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25 (NIPS) , pp. 1097-1105, 2012.
  12. Fengfu Li and Bin Liu. Ternary Weight Networks. CoRR , abs/1605.04711, 2016. · arXiv
  13. Naveen Mellempudi, Abhisek Kundu, Dheevatsa Mudigere, Dipankar Das, Bharat Kaul, and Pradeep Dubey. Ternary Neural Networks with Fine-Grained Quantization. CoRR , abs/1705.01462, 2017. · arXiv
  14. Asit K. Mishra, Jeffrey J. Cook, Eriko Nurvitadhi, and Debbie Marr. WRPN: Training and Inference using Wide Reduced-Precision Networks. CoRR , abs/1704.03079, 2017. · arXiv
  15. Daisuke Miyashita, Edward H. Lee, and Boris Murmann. Convolutional Neural Networks using Logarithmic Data Representation. CoRR , abs/1603.01025, 2016. · arXiv
  16. Vinod Nair and Geoffrey E. Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. 27th International Conference on Machine Learning (ICML) , pp. 807-814, 2010.
  17. Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning , number 2, pp. 5, 2011.
  18. Eunhyeok Park, Junwhan Ahn, and Sungjoo Yoo. Weighted-Entropy-Based Quantization for Deep Neural Networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , July 2017.
  19. Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. CoRR , abs/1603.05279, 2016. · arXiv
  20. Swagath Venkataramani, Jungwook Choi, Vijayalakshmi Srinivasan, Kailash Gopalakrishnan, and Leland Chang. POSTER: Design Space Exploration for Performance Optimization of Deep Neural Networks on Shared Memory Accelerators. Proc. PACT 2017 .
  21. Ying Zhang, Mohammad Pezeshki, Philemon Brakel, Saizheng Zhang, César Laurent, Yoshua Bengio, and Aaron C. Courville. Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks. CoRR , abs/1701.02720, 2017. · arXiv
  22. Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients. CoRR , abs/1606.06160, 2016. · arXiv
  23. Shuchang Zhou, Yuzhi Wang, He Wen, Qinyao He, and Yuheng Zou. Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks. CoRR , abs/1706.07145, 2017. · arXiv
  24. Chenzhuo Zhu, Song Han, Huizi Mao, and William J. Dally. Trained Ternary Quantization. International Conference on Learning Representations (ICLR) , 2017.

Cited by

Sorted by citations

Authors
Choi, J., Wang, Z., Venkataramani, S., Chuang, P. I.-J., Srinivasan, V., & Gopalakrishnan, K.
Affiliations
IBM (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1805.06085
arXiv
1805.06085
OpenAlex
W2786771851
Cited by
703
References
4
Citations by year
2026: 2 · 2025: 35 · 2024: 109 · 2023: 124 · 2022: 94 · 2021: 156 · 2020: 124 · 2019: 45 · 2018: 14
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Numerical Methods and Algorithms
Keywords
activation quantization, neural network quantization, low-bit precision, 4-bit quantization, weight quantization, inference acceleration, on-chip memory, hardware efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review