CoChat
Sign Up
Cited by 13Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

The DOI matches its Crossref record; the references it cites are not checked yet.

Bit-shrinking: Limiting Instantaneous Sharpness for Improving Post-training Quantization

Lin, Chen, Peng, Bo, Li, Zheyang, Tan, Wenming, Ren, Ye, Xiao, Jun, Pu, Shiliang·2023·2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr52729.2023.01554
Open this paper in CoChat — free account

Abstract

Post-training quantization (PTQ) is an effective compression method to reduce the model size and computational cost. However, quantizing a model into a low-bit one, e.g., lower than 4, is difficult and often results in non-negligible performance degradation. To address this, we investigate the loss landscapes of quantized networks with various bit-widths. We show that the network with more ragged loss surface, is more easily trapped into bad local minima, which mostly appears in low-bit quantization. A deeper analysis indicates, the ragged surface is caused by the injection of excessive quantization noise. To this end, we detach a sharpness term from the loss which reflects the impact of quantization noise. To smooth the rugged loss surface, we propose to limit the sharpness term small and stable during optimization. Instead of directly optimizing the target bit network, we design a self-adapted shrinking scheduler for the bit-width in continuous domain from high bit-width to the target by limiting the increasing sharpness term within a proper range. It can be viewed as iteratively adding small “instant” quantization noise and adjusting the network to eliminate its impact. Widely experiments including classification and detection tasks demonstrate the effectiveness of the Bit-shrinking strategy in PTQ. On the Vision Transformer models, our INT8 and INT6 models drop within 0.5% and 1.5% Top-1 accuracy, respectively. On the traditional CNN networks, our INT4 quantized models drop within 1.3% and 3.5% Top-1 accuracy on ResNet18 and MobileNetV2 without fine-tuning, which achieves the state-of-the-art performance.

Overview

Integrity signals

Registry

Crossref · 10.1109/cvpr52729.2023.01554

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

The citation lists haven’t been fetched yet

References 49 · Cited by 13

Research works printed but not linked: 39

  1. 10.1109/CVPR.2019.00881 · DOI · Paper page
  2. hubara, 2020, Improving post training neural quantization Layer-wise calibration and integer programming
  3. uhlich, 2020, Mixed precision dnns All you need is a good parametrization
  4. 10.1109/CVPR.2018.00460 · DOI · Paper page
  5. kim, 2022, Ctmq: Cyclic training of convolutional neural networks with multiple quantization steps, ArXiv Preprint
  6. 10.1109/CVPR.2017.422 · DOI
  7. rastegari, 0, X nor-net: Imagenet classification using binary convolutional neural networks, European Conference on Computer Vision, 525
  8. howard, 2017, Mobilenets: Efficient convolutional neural networks for mobile vision applications, ArXiv Preprint
  9. touvron, 0, Training data-efficient image transformers & distillation through attention, International Conference on Machine Learning, 10347
  10. hochreiter, 1994, Simplifying neural nets by discovering flat minima, Advances in neural information processing systems
  11. szegedy, 2015, Rethinking the inception architecture for computer vision
  12. cai, 2020, Zeroq A novel zero shot quantization framework
  13. banner, 2018, Aciq Analytical clipping for integer quantization of neural networks, abs 1810 5723
  14. li, 2016, Pruning filters for efficient convnets, ArXiv Preprint
  15. wei, 2022, Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization, ArXiv Preprint
  16. 10.1109/CVPR42600.2020.00206 · DOI
  17. 10.1145/3219819.3219970 · DOI
  18. li, 2021, Brecq: Pushing the limit of post-training quantization by block reconstruction, ArXiv Preprint
  19. li, 2018, Visualizing the loss landscape of neural nets, Advances in neural information processing systems, 31
  20. liu, 2021, Post-training quantization for vision transformer, Advances in neural information processing systems, 34, 28092
  21. 10.1109/CVPR.2018.00826 · DOI
  22. zhou, 2016, Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients, ArXiv Preprint
  23. nagel, 2020, Up or down? adaptive rounding for post-training quantization
  24. meller, 2019, Same same but different-recovering neural network quantization error through weight factorization
  25. zhuang, 2021, Effective training of convolutional neural networks with low-bitwidth weights and activations, IEEE Transactions on Pattern Analysis and Machine Intelligence
  26. lin, 2022, Uwc: Unit-wise calibration towards rapid network compression, ArXiv Preprint
  27. yuan, 2021, Ptq4vit: Post-training quantization framework for vision transformers, ArXiv Preprint
  28. 10.1109/CVPRW53098.2021.00339 · DOI
  29. liu, 2021, Sharpness-aware quantization for deep neural networks, ArXiv Preprint
  30. 10.1109/CVPR46437.2021.00806 · DOI
  31. lin, 2017, Focal loss for dense object detection
  32. zhao, 2019, Improving neural network quantization without retraining using outlier channel splitting, ICML
  33. peisong, 0, Towards accurate post-training network quantization via bit-split and stitching, Proceedings of the 37nd International Conference on Machine Learning (ICML), 243
  34. 10.1007/s10994-021-06053-z · DOI
  35. 10.1109/CVPR42600.2020.00801 · DOI
  36. foret, 2020, Sharpness-aware minimization for efficiently improving generalization, ArXiv Preprint
  37. 2020, OpenMMLab's Image Classification Toolbox and Benchmark[CP/OL]
  38. chen, 2019, MMDetection: Open mmlab detection toolbox and benchmark, ArXiv Preprint
  39. wu, 2020, Adversarial weight perturbation helps robust generalization, Advances in neural information processing systems, 33, 2958

Metadata

Authors
Lin, C., Peng, B., Li, Z., Tan, W., Ren, Y., Xiao, J., & Pu, S.
Affiliations
  • Hikvision Research Institute
  • Hikvision (China) (CN)
  • Zhejiang University (CN)
ORCID
Venue
2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2023-06
Pages
16196-16205
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr52729.2023.01554
OpenAlex
W4386065629
Cited by
13
References
49
Citations by year
2026: 2 · 2025: 6 · 2024: 4 · 2023: 1
FWCI
1.39
Type
conference paper
Language
en
Open access
closed
Licence
Locations
IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
post-training quantization, loss landscape, quantization noise, local minima, low-bit quantization, model compression, vision transformer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Bit-shrinking: Limiting Instantaneous Sharpness for Improving Post-training Quantization. https://app.cochat.ai/papers/10.1109/cvpr52729.2023.01554
The paper
Lin, C., Peng, B., Li, Z., Tan, W., Ren, Y., Xiao, J., & Pu, S. (2023). Bit-shrinking: Limiting Instantaneous Sharpness for Improving Post-training Quantization. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr52729.2023.01554