CoChat
Sign Up
PreprintCited by 36Advanced Neural Network ApplicationsNeural Networks and ApplicationsAdvanced Memory and Neural Computing

Training Binary Neural Networks through Learning with Noisy Supervision

Open access
Han, Kai, Wang, Yunhe, Xu, Yixing, Xu, Chunjing, Wu, Enhua, Xu, Chang·2020·arXiv
DOI 10.48550/arxiv.2010.04871arXiv 2010.04871
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper formalizes the binarization operations over neural networks from a learning perspective. In contrast to classical hand crafted rules (\eg hard thresholding) to binarize full-precision neurons, we propose to learn a mapping from full-precision neurons to the target binary ones. Each individual weight entry will not be binarized independently. Instead, they are taken as a whole to accomplish the binarization, just as they work together in generating convolution features. To help the training of the binarization mapping, the full-precision neurons after taking sign operations is regarded as some auxiliary supervision signal, which is noisy but still has valuable guidance. An unbiased estimator is therefore introduced to mitigate the influence of the supervision noise. Experimental results on benchmark datasets indicate that the proposed binarization technique attains consistent improvements over baselines.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2010.04871

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Combining k-means based vector quantization with pruning and Huffman coding can further reduce model size.

    “In addition to that, jointly applying k-means based vector quantization with pruning and Huffman coding can further reduce the model size [84].”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [84]

    The Introduction (p. 1) says compression methods, including pruning and quantization, reduce model parameters. The References (p. 9) list a work combining pruning, trained quantization, and Huffman coding, but this paper does not specify k-means-based vector quantization or state that jointly applying those three techniques further reduces model size. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

  • Quantization compresses a neural network by using lower bit-width for weight values without changing the model architecture, which is particularly useful for carefully designed network architectures like transformers.

    “Among various compression methods like pruning [16, 29, 19] and weight decomposition [36], quantization method [9, 37, 7, 27, 14] compresses a neural network by using lower bit-width for weight values without changing the model architecture, which is particularly useful for carefullydesigned network architectures like transformers.”

    From Post-Training Quantization for Vision Transformer (2021) · cites this paper as [14]

    The Introduction (p. 1) and §3.1 (p. 3) describe quantizing weights to one bit and reducing the storage needed for convolutional parameters. The paper does not establish that quantization leaves the architecture unchanged or that it is particularly useful for transformers; its discussion concerns convolutional networks. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. Bengio, Y., Leonard, N., and Courville, A. C. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv: Learning , 2013.
  2. Cai, Z., He, X., Sun, J., and Vasconcelos, N. Deep learning with low precision by half-wave gaussian quantization. In CVPR , 2017.
  3. Chen, H., Wang, Y., Xu, C., Yang, Z., Liu, C., Shi, B., Xu, C., Xu, C., and Tian, Q. Data-free learning of student networks. In ICCV , 2019.
  4. Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. L. Semantic image segmentation with deep convolutional nets and fully connected crfs. In ICLR , 2016.
  5. Courbariaux, M., Bengio, Y., and David, J.-P. Binaryconnect: Training deep neural networks with binary weights during propagations. In NeurIPS , pp. 3123-3131, 2015.
  6. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR , pp. 248-255. Ieee, 2009.
  7. Denton, E. L., Zaremba, W., Bruna, J., LeCun, Y., and Fergus, R. Exploiting linear structure within convolutional networks for efficient evaluation. In NeurIPS , pp. 1269-1277, 2014.
  8. Goldberger, J. and Ben-Reuven, E. Training deep neuralnetworks using a noise adaptation layer. In ICLR , 2017.
  9. Gong, R., Liu, X., Jiang, S., Li, T., Hu, P., Lin, J., Yu, F., and Yan, J. Differentiable soft quantization: Bridging fullprecision and low-bit neural networks. In ICCV , 2019.
  10. Gu, J., Li, C., Zhang, B., Han, J., Cao, X., Liu, J., and Doermann, D. Projection convolutional neural networks for 1-bit cnns via discrete back propagation. In AAAI , 2019.
  11. Gupta, S., Agrawal, A., Gopalakrishnan, K., and Narayanan, P. Deep learning with limited numerical precision. In ICML , pp. 1737-1746, 2015.
  12. Han, B., Yao, J., Niu, G., Zhou, M., Tsang, I., Zhang, Y., and Sugiyama, M. Masking: A new perspective of noisy supervision. In NeurIPS , 2018a.
  13. Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I., and Sugiyama, M. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS , 2018b.
  14. Han, K., Guo, J., Zhang, C., and Zhu, M. Attribute-aware attention model for fine-grained representation learning. In ACM MM , 2018c.
  15. Han, K., Wang, Y., Tian, Q., Guo, J., Xu, C., and Xu, C. Ghostnet: More features from cheap operations. In CVPR , 2020.
  16. Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR , 2016.
  17. He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV , 2015.
  18. Helwegen, K., Widdicombe, J., Geiger, L., Liu, Z., Cheng, K.-T., and Nusselder, R. Latent weights do not exist: Rethinking binarized neural network optimization. In NeurIPS , 2019.
  19. Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015. · arXiv
  20. Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017. · arXiv
  21. Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. Binarized neural networks. In NeurIPS , pp. 4107-4115, 2016.
  22. Jiang, L., Zhou, Z., Leung, T., Li, L.-J., and Fei-Fei, L. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML , 2018.
  23. Lee, K.-H., He, X., Zhang, L., and Yang, L. Cleannet: Transfer learning for scalable image classifier training with label noise. In CVPR , 2018.
  24. Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H. P. Pruning filters for efficient convnets. In ICLR , 2017.
  25. Lin, X., Zhao, C., and Pan, W. Towards accurate binary convolutional neural network. In NeurIPS , 2017.
  26. Liu, C., Ding, W., Hu, Y., Zhang, B., Liu, J., and Guo, G. Gbcns: Genetic binary convolutional networks for enhancing the performance of 1-bit dcnns. In AAAI , 2019.
  27. Liu, Z., Wu, B., Luo, W., Yang, X., Liu, W., and Cheng, K.T. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In ECCV , 2018.
  28. Natarajan, N., Dhillon, I. S., Ravikumar, P. K., and Tewari, A. Learning with noisy labels. In NeurIPS , 2013.
  29. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS , 2019.
  30. Patrini, G., Rozza, A., Krishna Menon, A., Nock, R., and Qu, L. Making deep neural networks robust to label noise: A loss correction approach. In CVPR , 2017.
  31. Qin, H., Gong, R., Liu, X., Shen, M., Wei, Z., Yu, F., and Song, J. Forward and backward information retention for accurate binary neural networks. In CVPR , 2020.
  32. Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV , pp. 525-542. Springer, 2016.
  33. Ren, S., He, K., Girshick, R., and Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. In NeurIPS , 2015.
  34. Shen, M., Han, K., Xu, C., and Wang, Y. Searching for accurate binary neural architectures. In ICCV Workshops , 2019.
  35. Shen, M., Liu, X., Gong, R., and Han, K. Balanced binary neural networks with gated residual. In ICASSP , 2020.
  36. Shu, H., Wang, Y., Jia, X., Han, K., Chen, H., Xu, C., Tian, Q., and Xu, C. Co-evolutionary compression for unpaired image translation. In ICCV , 2019.
  37. Tanaka, D., Ikami, D., Yamasaki, T., and Aizawa, K. Joint optimization framework for learning with noisy labels. In CVPR , 2018.
  38. Vahdat, A. Toward robustness against label noise in training deep discriminative neural networks. In NeurIPS , 2017.
  39. Wan, D., Shen, F., Liu, L., Zhu, F., Qin, J., Shao, L., and Tao Shen, H. Tbn: Convolutional neural network with ternary inputs and binary weights. In ECCV , 2018.
  40. Wang, Y., Liu, W., Ma, X., Bailey, J., Zha, H., Song, L., and Xia, S.-T. Iterative learning with open-set noisy labels. In CVPR , 2018.
  41. Wang, Y., Ma, X., Chen, Z., Luo, Y., Yi, J., and Bailey, J. Symmetric cross entropy for robust learning with noisy labels. In ICCV , 2019.
  42. Xiao, T., Xia, T., Yang, Y ., Huang, C., and Wang, X. Learning from massive noisy labeled data for image classification. In CVPR , 2015.
  43. Yang, J., Shen, X., Xing, J., Tian, X., Li, H., Deng, B., Huang, J., and Hua, X.-s. Quantization networks. In CVPR , 2019.
  44. Yu, X., Han, B., Yao, J., Niu, G., Tsang, I., and Sugiyama, M. How does disagreement help generalization against label corruption? In ICML , 2019.
  45. Zhang, Z. and Sabuncu, M. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurIPS , 2018.
  46. Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., and Zou, Y. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160 , 2016. · arXiv

Cited by

Sorted by citations

Authors
Han, K., Wang, Y., Xu, Y., Xu, C., Wu, E., & Xu, C.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
1
Pages
4017-4026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2010.04871
arXiv
2010.04871
OpenAlex
W3091991030
Cited by
36
References
4
Citations by year
2025: 3 · 2024: 4 · 2023: 7 · 2022: 9 · 2021: 11 · 2020: 1 · 2019: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Machine Learning
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
Keywords
binary neural networks, noisy supervision, unbiased estimators, weight quantization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review