CoChat
Sign Up
PreprintCited by 753Advanced Neural Network ApplicationsLow-power high-performance VLSI designNumerical Methods and Algorithms

Quantizing deep convolutional networks for efficient inference: A whitepaper

Open access
Krishnamoorthi, Raghuraman·2018·arXiv
DOI 10.48550/arxiv.1806.08342arXiv 1806.08342
Open this paper in CoChat — free accountOpen PDF

Abstract

We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks for a wide variety of CNN architectures. Model sizes can be reduced by a factor of 4 by quantizing weights to 8-bits, even when 8-bit arithmetic is not supported. This can be achieved with simple, post training quantization of weights.We benchmark latencies of quantized networks on CPUs and DSPs and observe a speedup of 2x-3x for quantized implementations compared to floating point on CPUs. Speedups of up to 10x are observed on specialized processors with fixed point SIMD capabilities, like the Qualcomm QDSPs with HVX. Quantization-aware training can provide further improvements, reducing the gap to floating point to 1% at 8-bit precision. Quantization-aware training also allows for reducing the precision of weights to four bits with accuracy losses ranging from 2% to 10%, with higher accuracy drop for smaller networks.We introduce tools in TensorFlow and TensorFlowLite for quantizing convolutional networks and review best practices for quantization-aware training to obtain high accuracy with quantized weights and activations. We recommend that per-channel quantization of weights and per-layer quantization of activations be the preferred quantization scheme for hardware acceleration and kernel optimization. We also propose that future processors and hardware accelerators for optimized inference support precisions of 4, 8 and 16 bits.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1806.08342

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Weight-quantization into as few as 2-4 bits can often be tolerated without significant accuracy reduction.

    “Weight-quantization into as few as 2–4 bits can often be tolerated without significant accuracy reduction8,19,20.”

    From Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators (2023) · cites this paper as [19]

    Section 3.2.4 (p. 22) reports that, for most networks, fine-tuned 4-bit weights achieved accuracy within 5% of 8-bit quantization, with activations kept at 8 bits. This supports the 4-bit portion under those conditions, but the cited passages do not establish comparable accuracy for 2-bit weights. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Starting from a pre-trained FP32 model is common and recommended practice in QAT, improving accuracy and reducing required iterations.

    “It is common and recommended practice in the literature to start from a pre-trained FP32 model [12,15,17]. This approach not only improves accuracy but also reduces the number of required iterations.”

    From HADQ-Net: A Power-Efficient and Hardware-Adaptive Deep Convolutional Neural Network Translator Based on Quantization-Aware Training for Hardware Accelerators (2025) · cites this paper as [17]

    Section 3.2 (p. 15) recommends starting quantization-aware training from a pre-trained floating-point model, and the training comparison (p. 24) reports better accuracy from fine-tuning than from training from scratch. The passages do not establish that this approach reduces the number of required iterations, nor do they specify FP32 precision. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Researchers have proposed techniques for reducing transformers' size, latency and energy consumption for efficient inference for a wide range of applications.

    “These challenges have motivated researchers to propose techniques for reducing transformers' size, latency and energy consumption for efficient inference for a wide range of applications.”

    From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [44]

    The Introduction (pp. 3–4) describes research on reducing model size, speeding inference, and lowering power consumption across a range of models and use cases. But the paper concerns deep convolutional networks; these passages do not establish the claim specifically for transformers. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • The methods for reducing transformers' size, latency and energy consumption include pruning, quantization, knowledge distillation and neural architecture search.

    “The methods include pruning [40, 41], Quantization [42-44], Knowledge Distillation [45] and Neural Architecture Search [46].”

    From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [44]

    The citation is attached to quantization, and the Introduction (pp. 3–4) describes it as a way to reduce model size, speed inference, and lower power consumption. The paper studies convolutional networks, however, so it does not establish those benefits specifically for transformers. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Quantization allows for more power-efficient operations and convolution computations at the expense of lower bitwidth representation.

    “Quantization allows for more power-efficient operations and convolution computations at the expense of lower bitwidth representation.”

    From PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation (2021) · cites this paper as [5]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 5

  1. [3][3] F. N. Iandola, M. W. Moskewicz, K. Ashraf, S. Han, W. J. Dally, and K. Keutzer, 'Squeezenet: Alexnet-level accuracy with 50x fewer parameters and < 1mb model size,' CoRR , vol. abs/1602.07360, 2016. - · arXiv
  2. [6][6] S. Han, H. Mao, and W. J. Dally, 'Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding,' CoRR , vol. abs/1510.00149, 2015. - · arXiv
  3. [12][12] V. Sze, Y. Chen, T. Yang, and J. S. Emer, 'Efficient processing of deep neural networks: A tutorial and survey,' CoRR , vol. abs/1703.09039, 2017. - · arXiv
  4. [31][31] A. K. Mishra, E. Nurvitadhi, J. J. Cook, and D. Marr, 'WRPN: wide reducedprecision networks,' CoRR , vol. abs/1709.01134, 2017. - · arXiv
  5. [33][33] Y. He and S. Han, 'ADC: automated deep compression and acceleration with reinforcement learning,' CoRR , vol. abs/1802.03494, 2018. · arXiv

Cited by

Sorted by citations

Authors
Krishnamoorthi, R.
Corresponding author
Krishnamoorthi, Raghuraman
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1806.08342
arXiv
1806.08342
OpenAlex
W2809624076
Cited by
753
References
0
Citations by year
2026: 13 · 2025: 63 · 2024: 114 · 2023: 102 · 2022: 121 · 2021: 153 · 2020: 119 · 2019: 58 · 2018: 10
Type
posted content
Language
en
Open access
Licence
creativecommons.org/publicdomain/zero/1.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (public-domain)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Numerical Methods and Algorithms
Keywords
neural network quantization, convolutional neural networks, post-training quantization, quantization-aware training, integer arithmetic, model size reduction, inference latency, TensorFlow Lite, hardware acceleration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review