CoChat
Sign Up
PreprintCited by 126Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningStochastic Gradient Optimization Techniques

Micro-Batch Training with Batch-Channel Normalization and Weight Standardization

Open access
Qiao, Siyuan, Wang, Huiyu, Liu, Chenxi, Shen, Wei, Yuille, Alan·2019·arXiv
DOI 10.48550/arxiv.1903.10520arXiv 1903.10520
Open this paper in CoChat — free accountOpen PDF

Abstract

Batch Normalization (BN) has become an out-of-box technique to improve deep network training. However, its effectiveness is limited for micro-batch training, i.e., each GPU typically has only 1-2 images for training, which is inevitable for many computer vision tasks, e.g., object detection and semantic segmentation, constrained by memory consumption. To address this issue, we propose Weight Standardization (WS) and Batch-Channel Normalization (BCN) to bring two success factors of BN into micro-batch training: 1) the smoothing effects on the loss landscape and 2) the ability to avoid harmful elimination singularities along the training trajectory. WS standardizes the weights in convolutional layers to smooth the loss landscape by reducing the Lipschitz constants of the loss and the gradients; BCN combines batch and channel normalizations and leverages estimated statistics of the activations in convolutional layers to keep networks away from elimination singularities. We validate WS and BCN on comprehensive computer vision tasks, including image classification, object detection, instance segmentation, video recognition and semantic segmentation. All experimental results consistently show that WS and BCN improve micro-batch training significantly. Moreover, using WS and BCN with micro-batch training is even able to match or outperform the performances of BN with large-batch training.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1903.10520

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. [1][1] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, 'Semantic image segmentation with deep convolutional nets and fully connected crfs,' in International Conference on Learning Representations (ICLR) , 2015. -
  2. [3][3] S. Ioffe and C. Szegedy, 'Batch normalization: Accelerating deep network training by reducing internal covariate shift,' in Proceedings of the 32nd International Conference on Machine Learning (ICML) , 2015. [Online]. Available: http://jmlr.org/proceedings/ papers/v37/ioffe15.html -
  3. [4][4] K. He, X. Zhang, S. Ren, and J. Sun, 'Identity mappings in deep residual networks,' European Conference on Computer Vision (ECCV) , pp. 630-645, 2016. -
  4. [5][5] G. Huang, Z. Liu, and K. Q. Weinberger, 'Densely connected convolutional networks,' in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017, pp. 2261-2269. -
  5. [6][6] Y. Wu and K. He, 'Group normalization,' in European Conference on Computer Vision (ECCV) , 2018, pp. 3-19. -
  6. [7][7] J. L. Ba, J. R. Kiros, and G. E. Hinton, 'Layer normalization,' arXiv preprint arXiv:1607.06450 , 2016. - · arXiv
  7. [8][8] S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry, 'How does batch normalization help optimization?' in Advances in Neural Information Processing Systems (NeurIPS) , 2018, pp. 2488-2498. -
  8. [9][9] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, 'ImageNet Large Scale Visual Recognition Challenge,' International Journal of Computer Vision (IJCV) , vol. 115, no. 3, pp. 211-252, 2015. -
  9. [10][10] T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, and C. L. Zitnick, 'Microsoft COCO: common objects in context,' in European Conference on Computer Vision (ECCV) , 2014, pp. 740-755. -
  10. [11][11] A. Krizhevsky and G. Hinton, 'Learning multiple layers of features from tiny images,' Master's thesis, Department of Computer Science, University of Toronto , 2009. -
  11. [15][15] J. Long, E. Shelhamer, and T. Darrell, 'Fully convolutional networks for semantic segmentation,' in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2015, pp. 3431-3440. [Online]. Available: https://doi.org/10.1109/CVPR. 2015.7298965 - · DOI
  12. [16][16] S. Qiao, C. Liu, W. Shen, and A. L. Yuille, 'Few-shot image recognition by predicting parameters from activations,' in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. -
  13. [17][17] S. Qiao, W. Shen, Z. Zhang, B. Wang, and A. Yuille, 'Deep co-training for semi-supervised image recognition,' in European Conference on Computer Vision (ECCV) , 2018. -
  14. [18][18] W. Qiu, F. Zhong, Y. Zhang, S. Qiao, Z. Xiao, T. S. Kim, and Y. Wang, 'Unrealcv: Virtual worlds for computer vision,' in Proceedings of the 25th ACM international conference on Multimedia . ACM, 2017, pp. 1221-1224. -
  15. [19][19] K. Simonyan and A. Zisserman, 'Very deep convolutional networks for large-scale image recognition,' in International Conference on Learning Representations (ICLR) , 2015. -
  16. [20][20] Y. Wang, L. Xie, C. Liu, S. Qiao, Y. Zhang, W. Zhang, Q. Tian, and A. Yuille, 'SORT: Second-Order Response Transform for Visual Recognition,' IEEE International Conference on Computer Vision , 2017. -
  17. [21][21] Y. Wang, L. Xie, S. Qiao, Y. Zhang, W. Zhang, and A. L. Yuille, 'Multi-scale spatially-asymmetric recalibration for image classification,' in Proceedings of the European Conference on Computer Vision (ECCV) , 2018, pp. 509-525. -
  18. [22][22] C. Yang, L. Xie, S. Qiao, and A. Yuille, 'Knowledge distillation in generations: More tolerant teachers educate better students,' AAAI , 2018. -
  19. [23][23] Z. Zhang, S. Qiao, C. Xie, W. Shen, B. Wang, and A. L. Yuille, 'Single-shot object detection with enriched semantics,' in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018, pp. 5813-5821. -
  20. [24][24] X. Glorot and Y. Bengio, 'Understanding the difficulty of training deep feedforward neural networks,' in Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS) , 2010, pp. 249-256. -
  21. [25][25] K. He, X. Zhang, S. Ren, and J. Sun, 'Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,' in IEEE International Conference on Computer Vision (ICCV) , 2015, pp. 1026-1034. -
  22. [26][26] D. Ulyanov, A. Vedaldi, and V. Lempitsky, 'Instance normalization: The missing ingredient for fast stylization,' arXiv preprint arXiv:1607.08022 , 2016. - · arXiv
  23. [27][27] T. Salimans and D. P. Kingma, 'Weight normalization: A simple reparameterization to accelerate training of deep neural networks,' in Advances in Neural Information Processing Systems (NeurIPS) , 2016, pp. 901-909. -
  24. [28][28] L. Huang, X. Liu, Y. Liu, B. Lang, and D. Tao, 'Centered weight normalization in accelerating training of deep neural networks,' in Proceedings of the IEEE International Conference on Computer Vision , 2017. -
  25. [29][29] A. E. Orhan and X. Pitkow, 'Skip connections eliminate singularities,' International Conference on Learning Representations (ICLR) , 2018. -
  26. [30][30] H. Wei, J. Zhang, F. Cousseau, T. Ozeki, and S.-i. Amari, 'Dynamics of learning near singularities in layered networks,' Neural computation , vol. 20, no. 3, pp. 813-843, 2008. -
  27. [32][32] A. S. Morcos, D. G. Barrett, N. C. Rabinowitz, and M. Botvinick, 'On the importance of single directions for generalization,' arXiv preprint arXiv:1803.06959 , 2018. - · arXiv
  28. [33][33] P. Luo, X. Wang, W. Shao, and Z. Peng, 'Towards understanding regularization in batch normalization,' in International Conference on Learning Representations (ICLR) , 2019. -
  29. [34][34] J. Kohler, H. Daneshmand, A. Lucchi, M. Zhou, K. Neymeyr, and T. Hofmann, 'Towards a theoretical understanding of batch normalization,' arXiv preprint arXiv:1805.10694 , 2018. - · arXiv
  30. [35][35] G. Yang, J. Pennington, V. Rao, J. Sohl-Dickstein, and S. S. Schoenholz, 'A mean field theory of batch normalization,' in International Conference on Learning Representations (ICLR) , 2019. -
  31. [36][36] S. Arora, Z. Li, and K. Lyu, 'Theoretical analysis of auto ratetuning by batch normalization,' in International Conference on Learning Representations (ICLR) , 2019. -
  32. [37][37] P.-A. Absil, R. Mahony, and R. Sepulchre, Optimization algorithms on matrix manifolds . Princeton University Press, 2009. -
  33. [38][38] M. Cho and J. Lee, 'Riemannian approach to batch normalization,' in Advances in Neural Information Processing Systems , 2017, pp. 5225-5235. -
  34. [39][39] P. Luo, J. Ren, and Z. Peng, 'Differentiable learning-to-normalize via switchable normalization,' arXiv preprint arXiv:1806.10779 , 2018. - · arXiv
  35. [40][40] C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L. Li, L. Fei-Fei, A. L. Yuille, J. Huang, and K. Murphy, 'Progressive neural architecture search,' in European Conference on Computer Vision (ECCV) , 2018, pp. 19-35. [Online]. Available: https: //doi.org/10.1007/978-3-030-01246-5 2 - · DOI
  36. [41][41] W. Shao, T. Meng, J. Li, R. Zhang, Y. Li, X. Wang, and P. Luo, 'Ssn: Learning sparse switchable normalization via sparsestmax,' arXiv preprint arXiv:1903.03793 , 2019. - · arXiv
  37. [42][42] P. Luo, P. Zhanglin, S. Wenqi, Z. Ruimao, R. Jiamin, and W. Lingyun, 'Differentiable dynamic normalization for learning deep representation,' in International Conference on Machine Learning , 2019, pp. 4203-4211. -
  38. [43][43] Y. Nesterov, Introductory lectures on convex optimization: A basic course . Springer Science & Business Media, 2013, vol. 87. -
  39. [44][44] S. Bubeck et al. , 'Convex optimization: Algorithms and complexity,' Foundations and Trends R © in Machine Learning , vol. 8, no. 3-4, pp. 231-357, 2015. -
  40. [45][45] V. Nair and G. E. Hinton, 'Rectified linear units improve restricted boltzmann machines,' in Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel , 2010. [Online]. Available: http: //www.icml2010.org/papers/432.pdf -
  41. [46][46] I. Gitman and B. Ginsburg, 'Comparison of batch normalization and weight normalization algorithms for the large-scale image classification,' arXiv preprint arXiv:1709.08145 , 2017. - · arXiv
  42. [47][47] S. Ioffe, 'Batch renormalization: Towards reducing minibatch dependence in batch-normalized models,' in Advances in Neural Information Processing Systems (NeurIPS) , 2017, pp. 1945-1953. -
  43. [48][48] S. Xie, R. B. Girshick, P. Doll´ ar, Z. Tu, and K. He, 'Aggregated residual transformations for deep neural networks,' in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017, pp. 5987-5995. [Online]. Available: http://arxiv.org/abs/ 1611.05431 -
  44. [50][50] K. He, G. Gkioxari, P. Doll´ ar, and R. Girshick, 'Mask r-cnn,' in IEEE International Conference on Computer Vision (ICCV) , 2017, pp. 2961-2969. -
  45. [52][52] S. Ren, K. He, R. B. Girshick, and J. Sun, 'Faster R-CNN: towards real-time object detection with region proposal networks,' in Advances in Neural Information Processing Systems (NeurIPS) , 2015, pp. 91-99. [Online]. Available: http://papers.nips.cc/paper/ 5638-faster-r-cnn-towards-real-time-object-detection-with-region-proposal-net -
  46. [53][53] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, 'Rethinking atrous convolution for semantic image segmentation,' arXiv preprint arXiv:1706.05587 , 2017. - · arXiv
  47. [54][54] B. Hariharan, P. Arbelaez, L. D. Bourdev, S. Maji, and J. Malik, 'Semantic contours from inverse detectors,' in IEEE International Conference on Computer Vision (ICCV) , 2011, pp. 991-998. [Online]. Available: https://doi.org/10.1109/ICCV.2011.6126343 - · DOI
  48. [55][55] J. Lin, C. Gan, and S. Han, 'Temporal shift module for efficient video understanding,' arXiv preprint arXiv:1811.08383 , 2018. - · arXiv
  49. [56][56] B. Zhou, A. Andonian, A. Oliva, and A. Torralba, 'Temporal relational reasoning in videos,' in European Conference on Computer Vision (ECCV) , 2018, pp. 803-818. -
  50. [57][57] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev et al. , 'The kinetics human action video dataset,' arXiv preprint arXiv:1705.06950 , 2017. <!-- image --> A Siyuan Qiao received B.E. in Computer Science at Shanghai Jiao Tong University in 2016. He is currently a Ph.D. student at Johns Hopkins University, where he is advised by Bloomberg Distinguished Professor Alan Yuille. From June 2017 to August 2017, he worked at Baidu IDL as an intern. He interned at Adobe Inc. from June 2018 to August 2018. He has also spent time at University of California, Los Angeles, and YITU Technology. His research interests are computer vision and deep learning. <!-- image --> <!-- image --> <!-- image --> Huiyu Wang is a Ph.D. student in Computer Science at Johns Hopkins University, advised by Bloomberg Distinguished Professor Alan Yuille. He received M.S. in Electrical Engineering at University of California, Los Angeles in 2017 and B.S. in Information Engineering at Shanghai Jiao Tong University in 2015. He also spent wonderful summers at Google, Allen Institute for Artificial Intelligence(AI2), and TuSimple. His research interests are computer vision and machine learning. Chenxi Liu is a Ph.D. student at Johns Hopkins University, where his advisor is Bloomberg Distinguished Professor Alan Yuille. Before that, he received M.S. at University of California, Los Angeles and B.E. at Tsinghua University. He has also spent time at Facebook, Google, Adobe, Toyota Technological Institute at Chicago, University of Toronto, and Rice University. His research lies in computer vision and natural language processing. Wei Shen received his B.S. and Ph.D. degree both in Electronics and Information Engineering from the Huazhong University of Science and Technology, Wuhan, China, in 2007 and in 2012. From April 2011 to November 2011, he worked in Microsoft Research Asia as an intern. In 2012, he joined the School of Communication and Information Engineering, Shanghai University and served as an assistant and associate professor until Oct 2018. He is currently an Assistant Research Professor at the Department of Computer Science, Johns Hopkins University. His current research interests include computer vision, deep learning and biomedical image analysis. <!-- image --> Alan Yuille received his B.A. in mathematics from the University of Cambridge in 1976, and completed his Ph.D. in theoretical physics at Cambridge in 1980. He then held a postdoctoral position with the Physics Department, University of Texas at Austin, and the Institute for Theoretical Physics, Santa Barbara. He then became a research scientists at the Artificial Intelligence Laboratory at MIT (1982-1986) and followed this with a faculty position in the Division of Applied Sciences at Harvard (1986-1995), rising to the position of associate professor. From 1995-2002 he worked as a senior scientist at the Smith-Kettlewell Eye Research Institute in San Francisco. From 2002-2016 he was a full professor in the Department of Statistics at UCLA with joint appointments in Psychology, Computer Science, and Psychiatry. In 2016 he became a Bloomberg Distinguished Professor in Cognitive Science and Computer Science at Johns Hopkins University. He has won a Marr prize, a Helmholtz prize, and is a Fellow of IEEE. · arXiv

Cited by

Sorted by citations

Authors
Qiao, S., Wang, H., Liu, C., Shen, W., & Yuille, A.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1903.10520
arXiv
1903.10520
OpenAlex
W3048030262
Cited by
126
References
7
Citations by year
2025: 11 · 2024: 23 · 2023: 26 · 2022: 24 · 2021: 21 · 2020: 16 · 2019: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
Keywords
batch normalization, weight standardization, Lipschitz constant, object detection, semantic segmentation, instance segmentation, video recognition, image classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review