The DOI matches its DataCite record; 23 of the 23 references it cites were found in a registry or an index.
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Open accessAbstract
Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization. It also acts as a regularizer, in some cases eliminating the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.9% top-5 validation error (and 4.8% test error), exceeding the accuracy of human raters.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“This method ensures the performance of the output activations [81]. This performance follows a unit Gaussian distribution.”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [81]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Since the addition of three branches introduces the covariate shift issue (Ioffe and Szegedy 2015)”
From Make RepVGG Greater Again: A Quantization-Aware Approach (2024) · cites this paper as Ioffe and Szegedy 2015
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SGD can be applied naively to the federated optimization problem, where a single batch gradient calculation (say on a randomly selected client) is done per round of communication. This approach is computationally efficient, but requires very large numbers of rounds of training to produce good models (e.g., even using an advanced approach like batch normalization, Ioffe and Szegedy [21] trained MNI”
From Communication-Efficient Learning of Deep Networks from Decentralized Data (2016) · cites this paper as [21]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It mitigates the internal covariate shift that occurs as the distribution of each hidden layer's inputs changes during the training (as the parameters of the preceding layers change) and slows down the learning process.”
From Empirical asset pricing via machine learning: evidence from the European stock market (2021) · cites this paper as Ioffe and Szegedy (2015)
The cited source says what the claim attributes to it.
In the citing paper
“Second, we adopt the batch normalization algorithm introduced by Ioffe and Szegedy (2015).”
From Empirical asset pricing via machine learning: evidence from the European stock market (2021) · cites this paper as Ioffe and Szegedy (2015)
The cited source says what the claim attributes to it.
In the citing paper
“It occurs throughout training, because the weight updates to deeper layers result in a continuously changing distribution of signal at higher layers, which hinders the convergence of their weights.”
From Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation (2017) · cites this paper as Ioffe and Szegedy, 2015
The cited source says what the claim attributes to it.
In the citing paper
“Batch normalization standardizes intermediate outputs of the network to zero mean and unit variance for a batch of training examples [Ioffe and Szegedy, 2015].”
From Deep learning with convolutional neural networks for EEG decoding and visualization (2017) · cites this paper as Ioffe and Szegedy, 2015
The cited source says what the claim attributes to it.
In the citing paper
“Batch normalization is a type of supplemental layer which adaptively normalizes the input values of the following layer, mitigating the risk of overfitting, as well as improving gradient flow through the network, allowing higher learning rates, and reducing the dependence on initialization [28].”
From Convolutional neural networks: an overview and application in radiology (2018) · cites this paper as [28]
The cited source says what the claim attributes to it.
In the citing paper
“Unlike BatchNorm [Ioffe and Szegedy, 2015], commonly used in CNNs, LayerNorm cannot be folded into the previous layer due to its dynamic computational property, so we have to quantize it separately.”
From FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer (2022) · cites this paper as [Ioffe and Szegedy, 2015]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, in integer-only quantization, all the operations are performed using low-precision integer arithmetic [113,132,154,193,267], as illustrated in Figure 6 (Right).This permits the entire inference to be carried out with efficient integer arithmetic, without any floating point dequantization of any parameters or activations.”
From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [113]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“CMSIS-NN [136] is a library from ARM that helps quantizing and deploying NN models onto the ARM Cortex-M cores.Specifically, the library leverages fixed-point quantization [113,154,267] with power-oftwo scaling factors so that quantization and dequantization processes can be carried out efficiently with bit shifting operations.”
From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [113]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“as neural networks have the capability of learning directly from raw data [49].”
From Investigating Enhancements to Contrastive Predictive Coding for Human Activity Recognition (2023) · cites this paper as [49]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Recent advances in light-weight neural network architectures [15,13,14,9,11,10,12] are mostly based on the metric of FLOPs and do not consider these properties above.”
From ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design (2018) · cites this paper as [13]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Following the common practice [15,13,14], all networks in comparison have four levels of computational complexity, i.e. about 40, 140, 300 and 500+ MFLOPs.”
From ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design (2018) · cites this paper as [13]
Experimental — accuracy not yet measuredIn the citing paper
“Our reproduced GoogleNet has center crop top-1 error of 26.4%, close to that reported in [32] (25.2%).”
From Object Detection Networks on Convolutional Feature Maps (2017) · cites this paper as [32]
In the citing paper
“To avoid overfitting, both dropout [116] and batch normalization [33] can help to achieve better regularization.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
(2014) · Cited by 33,419
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
(2010) · Cited by 14,134
(2010) · International Conference on Machine Learning · Cited by 13,995
Independent component analysis: algorithms and applications
Hyvärinen, A., & Erkki Oja, A. (2000) · Neural Networks · doi:10.1016/s0893-6080(00)00026-5 · Cited by 8,926
(2010) · Cited by 8,261
On the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
(2013) · Cited by 3,740
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,202
Flagged first, then by citations
Cited by
Sorted by citations
Items 1,301–1,339 of 1,339
DeepFT: Fault-Tolerant Edge Computing using a Self-Supervised Deep Surrogate Model
Tuli, S., Casale, G., Cherkasova, L., & Jennings, N. R. (2023) · IEEE INFOCOM 2023 - IEEE Conference on Computer Communications · doi:10.1109/infocom53939.2023.10229049 · Cited by 23
Machine learning at the edge for AI-enabled multiplexed pathogen detection
Ganjalizadeh, V., Meena, G. G., Stott, M. A., Hawkins, A. R., & Schmidt, H. (2023) · Scientific Reports · doi:10.1038/s41598-023-31694-6 · Cited by 24
Recognizing Human Races through Machine Learning—A Multi-Network, Multi-Features Study
Darabant, A. S., Borza, D., & Danescu, R. (2021) · Mathematics · doi:10.3390/math9020195 · Cited by 24
Efficient and Lightweight Framework for Real-Time Ore Image Segmentation Based on Deep Learning
Sun, G., Huang, D., Cheng, L., Jia, J., Xiong, C., & Zhang, Y. (2022) · Minerals · doi:10.3390/min12050526 · Cited by 24
Lee, D., Lee, W. H., Seo, H. G., Oh, B.-M., Lee, J. C., & Kim, H. C. (2020) · IEEE Access · doi:10.1109/access.2020.3019532 · Cited by 22
A simulation-based few samples learning method for surface defect segmentation
Wei, T., Cao, D., Zheng, C., & Yang, Q. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.06.090 · Cited by 21
Lotfollahi, M., Wolf, F. A., & Theis, F. J. (2018) · doi:10.1101/478503 · Cited by 21
A comprehensive survey and taxonomy on privacy-preserving deep learning
Tran, A.-T., Luong, T.-D., & Huynh, V.-N. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127345 · Cited by 21
Siamese Networks for Weakly Supervised Human Activity Recognition
Sheng, T., & Huber, M. (2019) · 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC) · doi:10.1109/smc.2019.8914045 · Cited by 21
MaxDropout: Deep Neural Network Regularization Based on Maximum Output Values
do Santos, C. F. G., Colombo, D., Roder, M., & Papa, J. P. (2021) · 2020 25th International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr48806.2021.9412733 · Cited by 20
Learning Invariant Representations of Social Media Users
Andrews, N., & Bishop, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1178 · Cited by 19
Deep learning for Amazon satellite image analysis
Bragilevsky, L., & Bajic, I. V. (2017) · 2017 IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PACRIM) · doi:10.1109/pacrim.2017.8121895 · Cited by 17
Attacking mouse dynamics authentication using novel Wasserstein conditional DCGAN
Roy, A., Wong, K., & Phan, R. C.-W. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3245418 · Cited by 16
Transfer Learning Gaussian Anomaly Detection by Fine-tuning Representations
Rippel, O., Chavan, A., Lei, C., & Merhof, D. (2022) · Proceedings of the 2nd International Conference on Image Processing and Vision Engineering · doi:10.5220/0011063900003209 · Cited by 17
Weakly Supervised Learning of Placental Ultrasound Images with Residual Networks
Qi, H., Collins, S., & Noble, A. (2017) · Communications in computer and information science · doi:10.1007/978-3-319-60964-5_9 · Cited by 17
Siraj, M. S., & Ahad, M. A. R. (2020) · 2020 Joint 9th International Conference on Informatics, Electronics & Vision (ICIEV) and 2020 4th International Conference on Imaging, Vision & Pattern Recognition (icIVPR) · doi:10.1109/icievicivpr48672.2020.9306630 · Cited by 15
Generalization to New Actions in Reinforcement Learning
Jain, A., Szot, A., & Lim, J. J. (2020) · arXiv · doi:10.48550/arxiv.2011.01928 · Cited by 14
“Thus, each frame of the video is encoded through a 7-layer convolutional encoder with batch norm (Ioffe & Szegedy, 2015).”
MethodVessel Classification through Convolutional Neural Networks using Passive Sonar Spectrogram Images
Cinelli, L., Chaves, G., & Lima, M. (2018) · Anais de XXXVI Simpósio Brasileiro de Telecomunicações e Processamento de Sinais · doi:10.14209/sbrt.2018.340 · Cited by 13
Convolutional neural network for voice disorders classification using kymograms
Kumar, S. P., Narayanan, N., Ramachandran, J., & Thangavel, B. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105159 · Cited by 12
aSGD: Stochastic Gradient Descent with Adaptive Batch Size for Every Parameter
Shi, H., Yang, N., Tang, H., & Yang, X. (2022) · Mathematics · doi:10.3390/math10060863 · Cited by 11
Continual Egocentric Activity Recognition With Foreseeable-Generalized Visual–IMU Representations
He, C., Cheng, S., Qiu, Z., Xu, L., Meng, F., Wu, Q., & Li, H. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3371975 · Cited by 11
Toward Real-World Voice Disorder Classification
Kuo, H.-C., Hsieh, Y.-P., Tseng, H.-H., Wang, C.-T., Fang, S.-H., & Tsao, Y. (2023) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2023.3270532 · Cited by 10
Li, C., Zhang, L., Zhuo, Z., Su, F., Li, H., Xu, S., Liu, Y., Zhang, Z., Xie, Y., Yu, X., et al. (2023) · Theranostics · doi:10.7150/thno.79362 · Cited by 10
Modeling Uncertainty by Learning a Hierarchy of Deep Neural Connections
Rohekar, R. Y., Gurwicz, Y., Nisimov, S., & Novik, G. (2019) · arXiv · doi:10.48550/arxiv.1905.13195 · Cited by 10
Augmentation of small training data using GANs for enhancing the performance of image classification
Hung, S.-K., & Gan, J. Q. (2021) · 2020 25th International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr48806.2021.9412399 · Cited by 10
Wang, Y., Zhang, X., Xie, L., Zhou, J., Su, H., Zhang, B., & Hu, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6910 · Cited by 8
Introspective Perception for Long-Term Aerial Telemanipulation With Virtual Reality
Lee, J., Balachandran, R., Kondak, K., Coelho, A., Stefano, M. D., Humt, M., Feng, J., Asfour, T., & Triebel, R. (2024) · IEEE Transactions on Field Robotics · doi:10.55417/fr.2023010 · Cited by 8
Xie, Y., Peng, J., & Zhou, Y. (2019) · doi:10.21203/rs.2.18936/v1 · Cited by 5
Sheng, T., & Huber, M. (2025) · Sensors · doi:10.3390/s25134032 · Cited by 5
Ge, S., Zhan, W., Li, J., Li, L., Dong, P., Li, X., Wang, C., Wang, C., & Gao, Y. (2025) · Sustainable Cities and Society · doi:10.1016/j.scs.2025.106701 · Cited by 4
Martin, K., Adam, O., Obin, N., & Dufour, V. (2022) · doi:10.1101/2022.02.19.481011 · Cited by 6
Quantize YOLOv3-tiny For 5-bit Hardware
Hua, Y., Yu, L., Meng, X., & Qin, Z. (2021) · 2021 4th International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE) · doi:10.1109/aemcse51986.2021.00214 · Cited by 1
Evaluation of Deep Species Distribution Models Using Environment and Co-occurrences
Deneu, B., Servajean, M., Botella, C., & Joly, A. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28577-7_18 · Cited by 8
Generating Fair Universal Representations using Adversarial Models
Kairouz, P., Liao, J., Huang, C., Vyas, M., Welfert, M., & Sankar, L. (2019) · arXiv · doi:10.48550/arxiv.1910.00411 · Cited by 19
Investigating Enhancements to Contrastive Predictive Coding for Human Activity Recognition
Haresamudram, H., Essa, I., & Ploetz, T. (2022) · arXiv · Cited by 27
LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
Maes, L., Lidec, Q. L., Scieur, D., LeCun, Y., & Balestriero, R. (2026) · arXiv · doi:10.48550/arxiv.2603.19312
Energy-based Generative Adversarial Network
Zhao, J., Mathieu, M., & LeCun, Y. (2016) · arXiv · doi:10.48550/arxiv.1609.03126 · Cited by 912
“For training both EBGANs and GANs for the grid search, we use the following setting: • Batch normalization (Ioffe & Szegedy, 2015) is applied after each weight layer, except for the generator output layer and the discriminator input layer (Radford et al., 2015).”
MethodTraining Deep Nets with Sublinear Memory Cost
Chen, T., Xu, B., Zhang, C., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1604.06174 · Cited by 561
Deep Clustering for Unsupervised Learning of Visual Features
Caron, M., Bojanowski, P., Joulin, A., & Douze, M. (2018) · arXiv · doi:10.48550/arxiv.1807.05520 · Cited by 533
Metadata
- Authors
- Ioffe, S., & Szegedy, C.
- Affiliations
- Google (United States) (US)
- ORCID
- Szegedy, Christian
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1502.03167
- OpenAlex
- W1836465849
- Cited by
- 27,818
- References
- 23
- Citations by year
- 2026: 152 · 2025: 1317 · 2024: 2504 · 2023: 3465 · 2022: 3990 · 2021: 4698 · 2020: 4028 · 2019: 3832 · 2018: 2356 · 2017: 1153 · 2016: 273 · 2015: 43
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- batch normalization, internal covariate shift, deep neural network training, training speed, learning rate, parameter initialization, regularization, dropout, image classification, ensemble learning, ImageNet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
