Rethinking Differentiable Search for Mixed-Precision Neural Networks
Open accessAbstract
Low-precision networks, with weights and activations quantized to low bit-width, are widely used to accelerate inference on edge devices. However, current solutions are uniform, using identical bit-width for all filters. This fails to account for the different sensitivities of different filters and is suboptimal. Mixed-precision networks address this problem, by tuning the bit-width to individual filter requirements. In this work, the problem of optimal mixed-precision network search (MPS) is considered. To circumvent its difficulties of discrete search space and combinatorial optimization, a new differentiable search architecture is proposed, with several novel contributions to advance the efficiency by leveraging the unique properties of the MPS problem. The resulting Efficient differentiable MIxed-Precision network Search (EdMIPS) method is effective at finding the optimal bit allocation for multiple popular networks, and can search a large model, e.g. Inception-V3, directly on ImageNet without proxy task in a reasonable amount of time. The learned mixed-precision networks significantly outperform their uniform counterparts.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A mixed-precision model quantization method has been proposed by leveraging NAS [190], where different bit-widths are assigned to different layers/channels.
“Recently, a mixed-precision model quantization method has been proposed by leveraging the power of NAS [190], where different bit-widths are assigned to different layers/channels.”
From Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things (2021) · cites this paper as [190]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · see citing works
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137 · see citing works
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392 · see citing works
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015 · see citing works
Least squares quantization in PCM
Lloyd, S. (1982) · IEEE Transactions on Information Theory · doi:10.1109/tit.1982.1056489 · Cited by 15,836 · see citing works
Cascade R-CNN: Delving Into High Quality Object Detection
Cai, Z., & Vasconcelos, N. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00644 · Cited by 7,370 · see citing works
Self-citationLearning Transferable Architectures for Scalable Image Recognition
Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00907 · Cited by 6,631 · see citing works
Neural Architecture Search with Reinforcement LearningPreprint
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 5,091 · see citing works
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408 · see citing works
DARTS: Differentiable architecture search
Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248 · see citing works
Binarized Neural NetworksPreprint
Hubara, I., Soudry, D., & El-Yaniv, R. (2016) · arXiv.org · doi:10.48550/arxiv.1602.02505 · Cited by 2,241 · see citing works
Quantizing for minimum distortion
Max, J. (1960) · IEEE Transactions on Information Theory · doi:10.1109/tit.1960.1057548 · Cited by 2,070 · see citing works
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., & Zou, Y. (2016) · arXiv · doi:10.48550/arxiv.1606.06160 · Cited by 1,690 · see citing works
An overview of bilevel optimization
Colson, B., Marcotte, P., & Savard, G. (2007) · Annals of Operations Research · doi:10.1007/s10479-007-0176-2 · Cited by 1,452 · see citing works
HAQ: Hardware-Aware Automated Quantization With Mixed Precision
Wang, K., Liu, Z., Lin, Y., Lin, J., & Han, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00881 · Cited by 978 · see citing works
Single Path One-Shot Neural Architecture Search with Uniform Sampling
Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., & Sun, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58517-4_32 · Cited by 882 · see citing works
LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks
Zhang, D., Yang, J., Ye, D., & Hua, G. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01237-3_23 · Cited by 714 · see citing works
PACT: Parameterized Clipping Activation for Quantized Neural Networks
Choi, J., Wang, Z., Venkataramani, S., Chuang, P. I.-J., Srinivasan, V., & Gopalakrishnan, K. (2018) · arXiv · doi:10.48550/arxiv.1805.06085 · Cited by 703 · see citing works
Fixed Point Quantization of Deep Convolutional Networks
Lin, D. D., Talathi, S. S., & Annapureddy, V. S. (2015) · arXiv · doi:10.48550/arxiv.1511.06393 · Cited by 684 · see citing works
Understanding and Simplifying One-Shot Architecture Search
Bender, G., Kindermans, P.-J., Zoph, B., Vasudevan, V., & Le, Q. V. (2018) · International Conference on Machine Learning · Cited by 558 · see citing works
SMASH: One-Shot Model Architecture Search through HyperNetworksPreprint
Brock, A., Lim, T., Ritchie, J. M., & Weston, N. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05344 · Cited by 500 · see citing works
Deep Learning with Low Precision by Half-Wave Gaussian Quantization
Cai, Z., He, X., Sun, J., & Vasconcelos, N. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.574 · Cited by 498 · see citing works
Self-citationHAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision
Dong, Z., Yao, Z., Gholami, A., Mahoney, M., & Keutzer, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00038 · Cited by 492 · see citing works
Image and Video Compression for Multimedia Engineering: Fundamentals, Algorithms, and Standards
Shi, Y. Q., & Sun, H. (1999) · Thammasat University Digital Collections · Cited by 272 · see citing works
Towards Effective Low-Bitwidth Convolutional Neural Networks
Zhuang, B., Shen, C., Tan, M., Liu, L., & Reid, I. D. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00826 · Cited by 261 · see citing works
Fixed-point feedforward deep neural network design using weights +1, 0, and −1
Hwang, K., & Sung, W. (2014) · SiPS ... design and implementation - IEEE Workshop on Signal Processing Systems · doi:10.1109/sips.2014.6986082 · Cited by 258 · see citing works
Fixed point optimization of deep convolutional neural networks for object recognition
Anwar, S., Hwang, K., & Sung, W. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178146 · Cited by 246 · see citing works
SMASH: One-Shot Model Architecture Search through HyperNetworks
Brock, A., Lim, T., Ritchie, J. M., & Weston, N. J. (2018) · International Conference on Learning Representations · Cited by 217 · see citing works
Simulation-based word-length optimization method for fixed-point digital signal processing systems
Sung, W., & Kum, K.-I. (1995) · IEEE Transactions on Signal Processing · doi:10.1109/78.476465 · Cited by 207 · see citing works
Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
Wu, B., Wang, Y., Zhang, P., Tian, Y., Vajda, P., & Keutzer, K. (2018) · arXiv · doi:10.48550/arxiv.1812.00090 · Cited by 186 · see citing works
Structured Binary Neural Networks for Image Recognition
Zhuang, B., Shen, C., Tan, M., Chen, P., Liu, L., & Reid, I. D. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01638-0 · Cited by 23 · see citing works
Structured Binary Neural Networks for Image RecognitionPreprint
Zhuang, B., Shen, C., Tan, M., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09934 · Cited by 7 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Zhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774 · see citing works
Neural Architecture Search Survey: A Hardware Perspective
Chitty-Venkata, K. T., & Somani, A. K. (2023) · ACM Computing Surveys · doi:10.1145/3524500 · Cited by 106 · see citing works
Quantformer: Learning Extremely Low-Precision Vision Transformers
Wang, Z., Wang, C., Xu, X., Zhou, J., & Lu, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3229313 · Cited by 22 · see citing works
- Authors
- Cai, Z., & Vasconcelos, N.
- Affiliations
- University of California San Diego (US)
- Venue
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2020-06
- Pages
- 2346-2355
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3034887213
- Cited by
- 119
- References
- 40
- Citations by year
- 2026: 5 · 2025: 14 · 2024: 26 · 2023: 28 · 2022: 21 · 2021: 22 · 2020: 3
- FWCI
- 5.71
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1637941, 1924937)
- Nvidia
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- differentiable architecture search, bit-width allocation, low-precision inference, quantization, InceptionV3, edge devices
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
