CoChat
Sign Up
Cited by 135Advanced Neural Network ApplicationsAdversarial Robustness in Machine LearningDomain Adaptation and Few-Shot Learning

Data-Free Network Quantization With Adversarial Knowledge Distillation

Open access
Choi, Yoojin, Choi, Jihwan, El-Khamy, Mostafa, Lee, Jungwon·2020·2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
DOI 10.1109/cvprw50498.2020.00363
Open this paper in CoChat — free accountOpen PDF

Abstract

Network quantization is an essential procedure in deep learning for development of efficient fixed-point inference models on mobile or edge platforms. However, as datasets grow larger and privacy regulations become stricter, data sharing for model compression gets more difficult and restricted. In this paper, we consider data-free network quantization with synthetic data. The synthetic data are generated from a generator, while no data are used in training the generator and in quantization. To this end, we propose data-free adversarial knowledge distillation, which minimizes the maximum distance between the outputs of the teacher and the (quantized) student for any adversarial samples from a generator. To generate adversarial samples similar to the original data, we additionally propose matching statistics from the batch normalization layers for generated data and the original data in the teacher. Furthermore, we show the gain of producing diverse adversarial samples by using multiple generators and multiple students. Our experiments show the state-of-the-art data-free model compression and quantization results for (wide) residual networks and MobileNet on SVHN, CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets. The accuracy losses compared to using the original datasets are shown to be very minimal.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvprw50498.2020.00363

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Clustering can help reduce information loss from quantization.

    “In addition to rule-based and optimization-based nonuniform quantization, clustering can also be beneficial to alleviate the information loss due to quantization.”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [38]

    The Constraints section discusses network quantization generally, and Section 4.2 reports results for data-free quantization and quantization-aware training. Neither addresses whether clustering alleviates information loss from quantization; searches for clustering and cluster-centroid quantization found no supporting passage. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 58

  1. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436-444, 2015.
  2. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
  3. Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pages 1135-1143, 2015.
  4. Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural net- works. In Advances in Neural Information Processing Sys- tems, pages 2074-2082, 2016.
  5. Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic net- work surgery for efficient DNNs. In Advances In Neural Information Processing Systems, pages 1379-1387, 2016.
  6. Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In In- ternational Conference on Machine Learning, pages 2498- 2507, 2017.
  7. Christos Louizos, Karen Ullrich, and Max Welling. Bayesian compression for deep learning. In Advances in Neural Infor- mation Processing Systems, pages 3290-3300, 2017.
  8. Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through L0 regularization. In International Conference on Learning Representations, 2018.
  9. Jonathan Frankle and Michael Carbin. The lottery ticket hy- pothesis: Finding sparse, trainable neural networks. In Inter- national Conference on Learning Representations, 2018.
  10. Bin Dai, Chen Zhu, Baining Guo, and David Wipf. Com- pressing neural networks using the variational information bottleneck. In International Conference on Machine Learn- ing, pages 1135-1144, 2018.
  11. Song Han, Huizi Mao, and William J Dally. Deep com- pression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. In International Conference on Learning Representations, 2016.
  12. Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression. In Inter- national Conference on Learning Representations, 2017.
  13. Eunhyeok Park, Junwhan Ahn, and Sungjoo Yoo. Weighted- entropy-based quantization for deep neural networks. In Pro- ceedings of the IEEE International Conference on Computer Vision, pages 7197-7205, 2017.
  14. Frederick Tung and Greg Mori. Deep neural network com- pression by in-parallel pruning-quantization. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 2018.
  15. Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Uni- versal deep neural network compression. IEEE Journal of Selected Topics in Signal Processing, 2020.
  16. Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-Net: Imagenet classification using binary convolutional neural networks. In Proceedings of the European Conference on Computer Vision, pages 525-542, 2016.
  17. Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160, 2016. · arXiv
  18. Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. In International Conference on Learning Representations, 2017.
  19. Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconce- los. Deep learning with low precision by half-wave Gaus- sian quantization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5918- 5926, 2017.
  20. Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. LQ-Nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the Euro- pean Conference on Computer Vision, pages 365-382, 2018.
  21. Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2704-2713, 2018.
  22. Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pages 8612-8620, 2019.
  23. Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco An- dreetto, and Hartwig Adam. MobileNets: Efficient convolu- tional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017. · arXiv
  24. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distill- ing the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. · arXiv
  25. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fit- Nets: Hints for thin deep nets. In International Conference on Learning Representations, 2015.
  26. Sergey Zagoruyko and Nikos Komodakis. Paying more at- tention to attention: Improving the performance of convolu- tional neural networks via attention transfer. In International Conference on Learning Representations, 2017.
  27. Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distil- lation for knowledge transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9163-9171, 2019.
  28. Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Dark- Rank: Accelerating deep metric learning via cross sample similarities transfer. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
  29. Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Rela- tional knowledge distillation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3967-3976, 2019.
  30. Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. In NeurIPS Workshop on Learning with Limited Data, 2017.
  31. Kartikeya Bhardwaj, Naveen Suda, and Radu Marculescu. Dream distillation: A data-independent model compression framework. In ICML Joint Workshop on On-Device Machine Learning and Compact Deep Neural Network Representa- tions (ODML-CDNNR), 2019.
  32. Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. In Proceed- ings of the IEEE International Conference on Computer Vi- sion, pages 3514-3522, 2019.
  33. Paul Micaelli and Amos J Storkey. Zero-shot knowledge transfer via adversarial belief matching. In Advances in Neural Information Processing Systems, pages 9547-9557, 2019.
  34. Sergey Zagoruyko and Nikos Komodakis. Wide residual net- works. In Proceedings of the British Machine Vision Confer- ence, pages 87.1-87.12, 2016.
  35. Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bis- sacco, Bo Wu, and Andrew Y Ng. Reading digits in natu- ral images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
  36. Hongxu Yin, Pavlo Molchanov, Zhizhong Li, Jose M Al- varez, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via DeepInversion. arXiv preprint arXiv:1912.08795, 2019. · arXiv
  37. Raghuraman Krishnamoorthi. Quantizing deep convolu- tional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342, 2018. · arXiv
  38. Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. Zero-shot knowledge distillation in deep networks. In In- ternational Conference on Machine Learning, pages 4743- 4751, 2019.
  39. Jaemin Yoo, Minyong Cho, Taebum Kim, and U Kang. Knowledge extraction with no observable data. In Advances in Neural Information Processing Systems, pages 2701- 2710, 2019.
  40. Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equal- ization and bias correction. In Proceedings of the IEEE Inter- national Conference on Computer Vision, pages 1325-1334, 2019.
  41. Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Ne- mirovski. Robust Optimization, volume 28. Princeton Uni- versity Press, 2009.
  42. Dimitris Bertsimas, David B Brown, and Constantine Cara- manis. Theory and applications of robust optimization. SIAM review, 53(3):464-501, 2011.
  43. Naveed Akhtar and Ajmal Mian. Threat of adversarial at- tacks on deep learning in computer vision: A survey. IEEE Access, 6:14410-14430, 2018.
  44. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learn- ing models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
  45. Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge Be- longie. Generative adversarial perturbations. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4422-4431, 2018.
  46. Huaxia Wang and Chun-Nam Yu. A direct approach to robust deep learning using adversarial networks. In International Conference on Learning Representations, 2019.
  47. Yunseok Jang, Tianchen Zhao, Seunghoon Hong, and Honglak Lee. Adversarial defense via learning to generate diverse attacks. In Proceedings of the IEEE International Conference on Computer Vision, pages 2740-2749, 2019.
  48. Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9446- 9454, 2018.
  49. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672-2680, 2014.
  50. Ian Goodfellow. NIPS 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00160, 2016. · arXiv
  51. Ishan Durugkar, Ian Gemp, and Sridhar Mahadevan. Genera- tive multi-adversarial networks. In International Conference on Learning Representations, 2017.
  52. Tu Nguyen, Trung Le, Hung Vu, and Dinh Phung. Dual dis- criminator generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2670-2680, 2017.
  53. Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adver- sarial nets (GANs). In International Conference on Machine Learning, pages 224-232, 2017.
  54. Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung. MGAN: Training generative adversarial nets with multiple generators. In International Conference on Learning Repre- sentations, 2018.
  55. Thomas M Cover and Joy A Thomas. Elements of Informa- tion Theory. John Wiley & Sons, 2012.
  56. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
  57. Yurii Nesterov. A method for unconstrained convex mini- mization problem with the rate of convergence O(1/k 2 ). In Doklady AN USSR, volume 269, pages 543-547, 1983.
  58. Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradi- ent descent with warm restarts. In International Conference on Learning Representations, 2017.

Cited by

Sorted by citations

Authors
Choi, Y., Choi, J., El-Khamy, M., & Lee, J.
Affiliations
  • Samsung (South Korea) (KR)
  • Daegu Gyeongbuk Institute of Science and Technology (KR)
ORCID
Venue
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
Publisher
IEEE
Published
2020-06
Pages
3047-3057
DOI
10.1109/cvprw50498.2020.00363
OpenAlex
W3020927126
Cited by
135
References
7
Citations by year
2026: 3 · 2025: 21 · 2024: 31 · 2023: 33 · 2022: 20 · 2021: 25 · 2020: 2
FWCI
6.17
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
data-free quantization, neural network quantization, model compression, synthetic data generation, adversarial examples, batch normalization statistics, ResNet, MobileNet

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review