An Empirical Model of Large-Batch Training
Open accessThe DOI matches its DataCite record; 55 of the 55 references it cites were found in a registry or an index.
Abstract
In an increasing number of domains it has been demonstrated that deep learning models can be trained using relatively large batch sizes without sacrificing data efficiency. However the limits of this massive data parallelism seem to differ from domain to domain, ranging from batches of tens of thousands in ImageNet to batches of millions in RL agents that play the game Dota 2. To our knowledge there is limited conceptual understanding of why these limits to batch size differ or how we might choose the correct batch size in a new domain. In this paper, we demonstrate that a simple and easy-to-measure statistic called the gradient noise scale predicts the largest useful batch size across many domains and applications, including a number of supervised learning datasets (MNIST, SVHN, CIFAR-10, ImageNet, Billion Word), reinforcement learning domains (Atari and Dota), and even generative model training (autoencoders on SVHN). We find that the noise scale increases as the loss decreases over a training run and depends on the model size primarily through improved model performance. Our empirically-motivated theory also describes the tradeoff between compute-efficiency and time-efficiency, and provides a rough model of the benefits of adaptive batch-size training.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“For language modeling, we train a size-2048 LSTM [HS97] on the One Billion Word Benchmark corpus [CMS + 13], using byte pair encoding (BPE) [SHB15] with a vocabulary of size 40,000 and a 512 -dimensional embedding space.”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“• ImageNet We train a size 50 ResNet [HZRS15] with Momentum on ImageNet [DDS + 09], and use a learning rate schedule that decays the learning rate three times during training.”
MethodLearning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196
Reading digits in natural images with unsupervised feature learning
Netzer, Y. (2024) · doi:10.57702/e7wwo0zk · Cited by 4,149
(2013) · Cited by 3,740
Large Scale GAN Training for High Fidelity Natural Image Synthesis
Brock, A., Donahue, J., & Simonyan, K. (2018) · arXiv · doi:10.48550/arxiv.1809.11096 · Cited by 3,717
Optimization Methods for Large-Scale Machine Learning
Bottou, L., Curtis, F. E., & Nocedal, J. (2018) · SIAM Review · doi:10.1137/16m1080173 · Cited by 3,128
ImageNet: A large-scale hierarchical image database
Deng, J. H., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161
InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.03657 · Cited by 1,156
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974 · cited 1×
“For language modeling, we train a size-2048 LSTM [HS97] on the One Billion Word Benchmark corpus [CMS + 13], using byte pair encoding (BPE) [SHB15] with a vocabulary of size 40,000 and a 512 -dimensional embedding space.”
MethodDistributed Prioritized Experience Replay
Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., van Hasselt, H., & Silver, D. (2018) · arXiv · doi:10.48550/arxiv.1803.00933 · Cited by 963
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2015) · arXiv · doi:10.48550/arxiv.1508.07909 · Cited by 950
TrueSkill™: A Bayesian Skill Rating System
Herbrich, R., Minka, T., & Graepel, T. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0076 · Cited by 639
One billion word benchmark for measuring progress in statistical language modeling
Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-564 · Cited by 627
Don't Decay the Learning Rate, Increase the Batch Size
Smith, S. L., Kindermans, P.-J., Ying, C., & Le, Q. V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00489 · Cited by 559
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., & Tang, P. T. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.04836 · Cited by 525
Large Batch Training of Convolutional Networks
You, Y., Gitman, I., & Ginsburg, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.03888 · Cited by 463
Deep Learning Scaling is Predictable, Empirically
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., & Zhou, Y. (2017) · arXiv · doi:10.48550/arxiv.1712.00409 · Cited by 427
You, Y., Zhang, Z., Hsieh, C.-J., Demmel, J., & Keutzer, K. (2018) · Proceedings of the 47th International Conference on Parallel Processing · doi:10.1145/3225058.3225069 · Cited by 384
Sample size selection in optimization methods for machine learning
Byrd, R. H., Chin, G. M., Nocedal, J., & Wu, Y. (2012) · Mathematical Programming · doi:10.1007/s10107-012-0572-5 · Cited by 369
Stochastic Gradient Descent as Approximate Bayesian Inference
Mandt, S., Hoffman, M. D., & Blei, D. M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.04289 · Cited by 369
Qualitatively characterizing neural network optimization problems
Goodfellow, I. J., Vinyals, O., & Saxe, A. M. (2014) · arXiv · doi:10.48550/arxiv.1412.6544 · Cited by 341
A Bayesian Perspective on Generalization and Stochastic Gradient DescentPreprint
Smith, S. L., & Le, Q. V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.06451 · Cited by 336
Jia, X., Song, S., He, W., Wang, Y., Rong, H., Zhou, F., Xie, L., Guo, Z., Yang, Y., Yu, L.-W., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.11205 · Cited by 281
Schaul, T., Zhang, S., & LeCun, Y. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.1106 · Cited by 266
Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
Akiba, T., Suzuki, S., & Fukuda, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.04325 · Cited by 260
Measuring the Effects of Data Parallelism on Neural Network Training
Shallue, C. J., Lee, J., Antognini, J., Sohl-Dickstein, J., Frostig, R., & Dahl, G. E. (2018) · arXiv · doi:10.48550/arxiv.1811.03600 · Cited by 229
Goh, G. (2017) · Distill · doi:10.23915/distill.00006 · Cited by 194
Understanding Short-Horizon Bias in Stochastic Meta-OptimizationPreprint
Wu, Y., Ren, M., Liao, R., & Grosse, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.02021 · Cited by 135
Gradient Diversity: a Key Ingredient for Scalable Distributed LearningPreprint
Yin, D., Pananjady, A., Lam, M. W. Y., Papailiopoulos, D., Ramchandran, K., & Bartlett, P. L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.05699 · Cited by 115
Ma, S., Bassily, R., & Belkin, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.06559 · Cited by 114
AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
Devarakonda, A., Naumov, M., & Garland, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.02029 · Cited by 97
On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient DescentPreprint
Golmant, N., Vemuri, N., Yao, Z., Feinberg, V., Gholami, A., Rothauge, K., Mahoney, M. W., & Gonzalez, J. E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.12941 · Cited by 96
Accelerated Methods for Deep Reinforcement Learning
Stooke, A., & Abbeel, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.02811 · Cited by 92
Gradient Descent Happens in a Tiny Subspace
Gur-Ari, G., Roberts, D. A., & Dyer, E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.04754 · Cited by 91
Coupling Adaptive Batch Sizes with Learning RatesPreprint
Balles, L., Romero, J., & Hennig, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.05086 · Cited by 86
Distributed Second-Order Optimization using Kronecker-Factored Approximations
Ba, J. L., Grosse, R., & Martens, J. (2017) · International Conference on Learning Representations · Cited by 66
Measuring the Intrinsic Dimension of Objective Landscapes
Li, C., Farkhoor, H., Liu, R., & Yosinski, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.08838 · Cited by 59
Big Batch SGD: Automated Inference using Adaptive Batch SizesPreprint
De, S., Yadav, A. K., Jacobs, D. W., & Goldstein, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.05792 · Cited by 56
Distributed Deep Reinforcement Learning: Learn How to Play Atari Games in 21 minutes
Adamski, I., Adamski, R., Grel, T., Jędrych, A., Kaczmarek, K., & Michalewski, H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-92040-5_19 · Cited by 40
Are Deep Policy Gradient Algorithms Truly Policy Gradient AlgorithmsPreprint
Ilyas, A., Engstrom, L., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., & Mądry, A. (2018) · arXiv (Cornell University) · Cited by 32
Large Scale Language Modeling: Converging on 40GB of Text in Four Hours
Puri, R., Kirby, R. M., Yakovenko, N., & Catanzaro, B. (2018) · International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD) · doi:10.1109/cahpc.2018.8645935 · Cited by 22
The Effect of Network Width on the Performance of Large-batch TrainingPreprint
Chen, L., Wang, H., Zhao, J., Papailiopoulos, D., & Koutris, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.03791 · Cited by 20
Large Scale Language Modeling: Converging on 40GB of Text in Four HoursPreprint
Puri, R., Kirby, R. M., Yakovenko, N., & Catanzaro, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.01371 · Cited by 15
Opening the Black Box of Deep Neural Networks via Information
Shwartz-Ziv, R., & Tishby, N. (2017) · arXiv · doi:10.48550/arxiv.1703.00810
Hoffer, E., Hubara, I., & Soudry, D. (2017) · arXiv · doi:10.48550/arxiv.1705.08741
Adding Gradient Noise Improves Learning for Very Deep Networks
Neelakantan, A., Vilnis, L., Le, Q. V., Sutskever, I., Kaiser, L., Kurach, K., & Martens, J. (2015) · arXiv · doi:10.48550/arxiv.1511.06807
Cited by
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Self-citationScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
“B crit is measured empirically from the data shown in Figure 18, but it is also roughly predicted by the gradient noise scale, as in [MKAT18]. and studying it as a function of N, D .”
BackgroundMethodInfluentialSelf-citationPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
“, better loss as a function of tokens seen) earlier in training, while larger batch sizes are beneficial later in training due to better gradient estimates (Smith et al., 2018; McCandlish et al., 2018), and (2) larger batch sizes result in larger matrix multiplication dimensions, which increases TPU efficiency.”
BackgroundImproved Denoising Diffusion Probabilistic Models
Nichol, A., & Dhariwal, P. (2021) · arXiv · doi:10.48550/arxiv.2102.09672 · Cited by 1,550
“We confirmed this by evaluating the gradient noise scales (McCandlish et al., 2018) for models trained with both objectives, as shown in Figure 7.”
MethodA Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
Metadata
- Authors
- McCandlish, S., Kaplan, J., Amodei, D., & Team, O. D.
- Affiliations
- Johns Hopkins University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1812.06162
- OpenAlex
- W2903697572
- Cited by
- 543
- References
- 54
- Citations by year
- 2026: 147 · 2025: 125 · 2024: 70 · 2023: 46 · 2022: 44 · 2021: 32 · 2020: 44 · 2019: 30 · 2018: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- large batch training, batch size selection, computational efficiency, time efficiency, adaptive batch size, deep learning training, reinforcement learning, generative model training, autoencoder, model size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.