The DOI matches its Crossref record; 252 of the 252 references it cites were found in a registry or an index.
Normalization Techniques in Training DNNs: Methodology, Analysis and Application
Open accessAbstract
Normalization techniques are essential for accelerating the training and improving the generalization of deep neural networks (DNNs), and have successfully been used in various applications. This paper reviews and comments on the past, present and future of normalization methods in the context of DNN training. We provide a unified picture of the main motivation behind different approaches from the perspective of optimization, and present a taxonomy for understanding the similarities and differences between them. Specifically, we decompose the pipeline of the most representative normalizing activation methods into three components: the normalization area partitioning, normalization operation and normalization representation recovery. In doing so, we provide insight for designing new normalization technique. Finally, we discuss the current progress in understanding normalization methods, and provide a comprehensive review of the applications of normalization for particular tasks, in which it can effectively solve the key issues.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–252 of 252
OOGAN: Disentangling GAN with One-Hot Sampling and Orthogonal Regularization
Liu, B., Zhu, Y., Fu, Z., de Melo, G., & Elgammal, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5919 · Cited by 39
How Does Batch Normalization Help Binary Training?Preprint
Sari, E., Belbahri, M., & Nia, V. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09139 · Cited by 38
Deep Isometric Learning for Visual RecognitionPreprint
Qi, H., You, C., Wang, X., Ma, Y., & Malik, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16992 · Cited by 38
Normalization Helps Training of Quantized LSTM
Hou, L., Zhu, J., Kwok, J. T.-Y., Gao, F., Qin, T., & Liu, T. (2019) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · Cited by 37
Li, X., Sun, W., & Wu, T. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_5 · Cited by 34
Local Context Normalization: Revisiting Local Normalization
Ortiz, A., Robinson, C., Morris, D., Fuentes, O., Kiekintveld, C. D., Hassan, M., & Jojić, N. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01129 · Cited by 33
Learning deep architectures via generalized Whitened Neural Networks
Luo, P. (2017) · International Conference on Machine Learning · Cited by 32
Accelerated gradient descent by factor-centering decomposition
Schraudolph, N. N. (1998) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-a-004263473 · Cited by 27
A Domain Agnostic Normalization Layer for Unsupervised Adversarial Domain Adaptation
Romijnders, R., Meletis, P., & Dubbelman, G. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00203 · Cited by 27
Network DeconvolutionPreprint
Ye, C., Evanusa, M., He, H. W., Mitrokhin, A., Goldstein, T., Yorke, J. A., Fermuller, C. M., & Aloimonos, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11926 · Cited by 27
Differentiable Dynamic Normalization for Learning Deep Representation
Luo, P., Peng, Z., Shao, W., Zhang, R., Ren, J., & Wu, L. (2019) · International Conference on Machine Learning · Cited by 26
Wang, C., Wu, Y., Vuong, Q., & Ross, K. W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02208 · Cited by 26
Group Whitening: Balancing Learning Efficiency and Representational Capacity
Huang, L., Zhou, Y., Liu, L., Zhu, F., & Shao, L. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00939 · Cited by 26
Self-citationRevisit Batch Normalization: New Understanding and Refinement via Composition Optimization
Lian, X., & Liu, J. (2019) · International Conference on Artificial Intelligence and Statistics · Cited by 25
IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization
Zhou, W., Lin, B. Y., & Ren, X. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17718 · Cited by 23
An Investigation Into the Stochasticity of Batch Whitening
Huang, L., Zhao, L., Zhou, Y., Zhu, F., Liu, L., & Shao, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00647 · Cited by 23
Self-citationLiao, Q., Kawaguchi, K., & Poggio, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.06160 · Cited by 22
Double Forward Propagation for Memorized Batch Normalization
Guo, Y. F., Wu, Q., Deng, C., Chen, J., & Tan, M. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11717 · Cited by 22
Kalman Normalization: Normalizing Internal Representations Across Network Layers
Wang, G., Peng, J., Luo, P., Wang, X., & Lin, L. (2018) · neural information processing systems · Cited by 22
Instance-Level Meta Normalization
Jia, S., Chen, D.-J., & Chen, H.-T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00500 · Cited by 22
Slowing Down the Weight Norm Increase in Momentum-based OptimizersPreprint
Heo, B., Chun, S., Oh, S. J., Han, D., Yun, S., Uh, Y., & Ha, J.-W. (2020) · arXiv (Cornell University) · Cited by 22
Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep LearningPreprint
Lubana, E. S., Dick, R. P., & Tanaka, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05956 · Cited by 22
Towards an Adversarially Robust Normalization ApproachPreprint
Awais, M., Shamshad, F., & Bae, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.11007 · Cited by 21
Understanding the Disharmony between Weight Normalization Family and Weight Decay
Li, X., Chen, S., & Yang, J. X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5904 · Cited by 21
DizzyRNN: Reparameterizing Recurrent Neural Networks for Norm-Preserving BackpropagationPreprint
Dorobantu, V. D., Stromhaug, P. A., & Renteria, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.04035 · Cited by 20
Chai, E., Pilancı, M., & Murmann, B. (2020) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/ieeeconf51394.2020.9443275 · Cited by 19
Exemplar Normalization for Learning Deep Representation
Zhang, R., Peng, Z., Wu, L., Li, Z., & Luo, P. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01274 · Cited by 18
Relative fisher information and natural gradient for learning large modular models
Sun, K., & Nielsen, F. (2017) · King Abdullah University of Science and Technology Repository (King Abdullah University of Science and Technology) · Cited by 17
CrossNorm: Normalization for Off-Policy TD Reinforcement LearningPreprint
Bhatt, A., Argus, M. J., Amiranashvili, A., & Brox, T. (2019) · arXiv (Cornell University) · Cited by 17
Li, W., Qi, K., Chen, W., & Zhou, Y. (2021) · IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA) · doi:10.1109/icaica52286.2021.9498199 · Cited by 17
Self-citationStochastic Normalizations as Bayesian Learning
Shekhovtsov, A., & Flach, B. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20890-5_30 · Cited by 16
Accelerated Large Batch Optimization of BERT Pretraining in 54 minutesPreprint
Zheng, S., Lin, H., Zha, S., & Li, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.13484 · Cited by 16
Learning Unitary Operators with Help From u(n)
Hyland, S. L., & Rätsch, G. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10928 · Cited by 14
Projection Based Weight Normalization for Deep Neural NetworksPreprint
Huang, L., Liu, X., Lang, B., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.02338 · Cited by 13
Self-citationWhitening and Coloring transform for GANs.Preprint
Siarohin, A., Sangineto, E., & Sebe, N. (2018) · arXiv (Cornell University) · Cited by 13
Layer-Wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs
Huang, L., Qin, J., Liu, L., Zhu, F., & Shao, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58536-5_23 · Cited by 13
Self-citationConvergence Analysis of Batch Normalization for Deep Neural Nets.Preprint
Ma, Y., & Klabjan, D. (2017) · arXiv (Cornell University) · Cited by 12
Rethinking Normalization and Elimination Singularity in Neural NetworksPreprint
Qiao, S., Wang, H., Liu, C., Shen, W., & Yuille, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09738 · Cited by 12
Towards Understanding Normalization in Neural ODEsPreprint
Gusak, J., Markeeva, L., Daulbaev, T., Katrutsa, A., Cichocki, A. S., & Oseledets, I. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.09222 · Cited by 12
Low-Precision Batch-Normalized ActivationsPreprint
Graham, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08231 · Cited by 11
Regularizing activations in neural networks via distribution matching with the Wasserstein metricPreprint
Joo, T., Kang, D., & Kim, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.05366 · Cited by 8
Training CNNs With Normalized Kernels
Özay, M., & Okatani, T. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11624 · Cited by 7
Spherical Motion Dynamics of Deep Neural Networks with Batch Normalization and Weight Decay.Preprint
Wan, R., Zhu, Z., Zhang, X., & Sun, J. (2020) · arXiv (Cornell University) · Cited by 7
Interpolating Convolutional Neural Networks Using Batch Normalization
Data, G. W. P., Ngu, K., Murray, D. W., & Prisacariu, V. A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01261-8_35 · Cited by 6
Unsupervised Batch Normalization
Kocyigit, M. T., Sevilla-Lara, L., Hospedales, T. M., & Bilen, H. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00467 · Cited by 6
Be Like Water: Robustness to Extraneous Variables Via Adaptive Feature NormalizationPreprint
Kaku, A., Mohan, S., Parnandi, A., Schambra, H. M., & Fernandez‐Granda, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04019 · Cited by 3
Spherical Perspective on Learning with Batch Norm.Preprint
Roburin, S., de Mont-Marin, Y. D., Bursuc, A., Marlet, R., Pérez, P., & Aubry, M. (2020) · arXiv (Cornell University) · Cited by 3
Optimal Quantization for Batch Normalization in Neural Network Deployments and BeyondPreprint
Lin, D., Sun, P., Xie, G., Zhou, S., & Zhang, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.13128 · Cited by 3
Finet: Using Fine-grained Batch Normalization to Train Light-weight Neural Networks
Luo, C., Zhan, J., Wang, L., & Gao, W. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534380 · Cited by 2
Unpaired Image Translation via Adaptive Convolution-based NormalizationPreprint
Cho, W., Kim, K., Kim, E., Kim, H. J., & Choo, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.13271 · Cited by 1
Hoffer, E., Hubara, I., & Soudry, D. (2017) · arXiv · doi:10.48550/arxiv.1705.08741
Improving robustness against common corruptions by covariate shift adaptation
Schneider, S., Rusak, E., Eck, L., Bringmann, O., Brendel, W., & Bethge, M. (2020) · arXiv · doi:10.48550/arxiv.2006.16971
Cited by
A review of convolutional neural networks in computer vision
Zhao, X., Wang, L., Zhang, Y., Han, X., Deveci, M., & Parmar, M. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10721-6 · Cited by 1,148
Survey on Machine Learning Biases and Mitigation Techniques
Siddique, S., Haque, M. A., George, R., Gupta, K. D., Gupta, D., & Faruk, M. J. H. (2023) · Digital · doi:10.3390/digital4010001 · Cited by 78
Lamaakal, I., Ouahbi, I., El Makkaoui, K., Maleh, Y., Pławiak, P., & Alblehai, F. (2024) · IEEE Access · doi:10.1109/access.2024.3406631 · Cited by 34
Metadata
- Authors
- Huang, L., Qin, J., Zhou, Y., Zhu, F., Liu, L., & Shao, L.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2023-08
- Volume
- 45
- Issue
- 8
- Pages
- 10173-10196
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4287660544
- Cited by
- 537
- References
- 252
- FWCI
- 45.07
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.