The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks
Open accessAbstract
Hardware-friendly network quantization (e.g., binary/uniform quantization) can efficiently accelerate the inference and meanwhile reduce memory consumption of the deep neural networks, which is crucial for model deployment on resource-limited devices like mobile phones. However, due to the discreteness of low-bit quantization, existing quantization methods often face the unstable training process and severe performance degradation. To address this problem, in this paper we propose Differentiable Soft Quantization (DSQ) to bridge the gap between the full-precision and low-bit networks. DSQ can automatically evolve during training to gradually approximate the standard quantization. Owing to its differentiable property, DSQ can help pursue the accurate gradients in backward propagation, and reduce the quantization loss in forward process with an appropriate clipping range. Extensive experiments over several popular network structures show that training low-bit neural networks with DSQ can consistently outperform state-of-the-art quantization methods. Besides, our first efficient implementation for deploying 2 to 4-bit DSQ on devices with ARM architecture achieves up to 1.7× speed up, compared with the open-source 8-bit high-performance inference framework NCNN [31].
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
(2017) · Cited by 11,625
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Identity Mappings in Deep Residual Networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1603.05027 · Cited by 10,665 · cited 2×
“For hyperparameters, we follow the rules described in the origin papers [7, 14, 15, 41].”
MethodHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408
Cited by
A Survey of Quantization Methods for Efficient Neural Network Inference
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2022) · Low-Power Computer Vision · doi:10.1201/9781003162810-13 · Cited by 1,219
“…models has also been shown as a promising method to recover the accuracy degradation after binarization/ternarization [33, 173, 189, 252]. c) Improved Training Method: Another interesting branch of work aims for better training methods for binary/ternary models [5, 20, 44, 73, 157, 160, 276, 279].”
Pruning and quantization for deep neural network acceleration: A survey
Liang, T., Glossner, J., Wang, L., Shi, S., & Zhang, X. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.07.045 · Cited by 943
Binary Neural Networks: A Survey
Qin, H., Gong, R., Liu, X., Bai, X., Song, J., & Sebe, N. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107281 · Cited by 546
“Following the same intuition, Gong et.al. present a Differential Soft Quantization (DSQ) method [63], replacing the traditional quantization function with a soft quantization function: where k determines the shape of the asymptotic function, s is a scaling factor to make the soft quantization function smooth and m i is…”
BackgroundMethodInfluentialSelf-citationForward and Backward Information Retention for Accurate Binary Neural Networks
Qin, H., Gong, R., Liu, X., Shen, M., Wei, Z., Yu, F., & Song, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00232 · Cited by 337
Self-citationToward Semantic Communications: Deep Learning-Based Image Semantic Coding
Huang, D., Gao, F., Tao, X., Du, Q., & Lu, J. (2023) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3221999 · Cited by 331
LSQ+: Improving low-bit quantization through learnable offsets and better initialization
Bhalgat, Y., Lee, J., Nagel, M., Blankevoort, T., & Kwak, N. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00356 · Cited by 236
BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
Li, Y., Gong, R., Tan, X., Yang, Y., Hu, P., Zhang, Q., Yu, F., Wang, W., & Gu, S. (2021) · arXiv · doi:10.48550/arxiv.2102.05426 · Cited by 190
Self-citation(2020) · neural information processing systems · Cited by 92
A Tiny CNN Architecture for Medical Face Mask Detection for Resource-Constrained Endpoints
Mohan, P., Paul, A. J., & Chirania, A. (2021) · Lecture Notes in Electrical Engineering · doi:10.1007/978-981-16-0749-3_52 · Cited by 70
Searching for Low-Bit Weights in Quantized Neural Networks
Yang, Z., Wang, Y., Han, K., Xu, C., Xu, C., Tao, D., & Xu, C. (2020) · arXiv · doi:10.48550/arxiv.2009.08695 · Cited by 64
QKD: Quantization-aware Knowledge Distillation
Kim, J., Bhalgat, Y., Lee, J., Patel, C., & Kwak, N. (2019) · arXiv · doi:10.48550/arxiv.1911.12491 · Cited by 44
Diffenderfer, J., & Kailkhura, B. (2021) · arXiv · doi:10.48550/arxiv.2103.09377 · Cited by 39
“DSQ (Gong et al., 2019) used differentiable soft quantization to have accurate gradients in backward propagation.”
MethodTraining Binary Neural Networks through Learning with Noisy Supervision
Han, K., Wang, Y., Xu, Y., Xu, C., Wu, E., & Xu, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04871 · Cited by 36
A Review of Recent Advances of Binary Neural Networks for Edge Computing
Zhao, W., Ma, T., Gong, X., Zhang, B., & Doermann, D. (2021) · IEEE Journal on Miniaturization for Air and Space Systems · doi:10.1109/jmass.2020.3034205 · Cited by 36
Distribution-aware Adaptive Multi-bit Quantization
Zhao, S., Yue, T., & Hu, X. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00916 · Cited by 33
Quantformer: Learning Extremely Low-Precision Vision Transformers
Wang, Z., Wang, C., Xu, X., Zhou, J., & Lu, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3229313 · Cited by 22
Yoo, J., & Ban, G. (2025) · Sensors · doi:10.3390/s25092918 · Cited by 10
A Novel Mixed-Precision Quantization Approach for CNNs
Wu, D., Wang, Y., Fei, Y., & Gao, G. (2025) · IEEE Access · doi:10.1109/access.2025.3551802 · Cited by 8
WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmetic
Ni, R., Chu, H.-M., Castañeda, O., Chiang, P.-Y., Studer, C., & Goldstein, T. (2020) · arXiv · doi:10.48550/arxiv.2007.13242 · Cited by 8
Oflamaz, C. U., & Yalçın, M. E. (2025) · Electronics · doi:10.3390/electronics14183686 · Cited by 4
Metadata
- Authors
- Gong, R., Liu, X., Jiang, S., Li, T., Hu, P., Lin, J., Yu, F., & Yan, J.
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 4851-4860
- OpenAlex
- W3004061291
- Cited by
- 466
- References
- 42
- FWCI
- 17.88
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.