The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
ImageNet Training in Minutes
Abstract
In this paper, we investigate large scale computers' capability of speeding up deep neural networks (DNN) training. Our approach is to use large batch size, powered by the Layer-wise Adaptive Rate Scaling (LARS) algorithm, for efficient usage of massive computing resources. Our approach is generic, as we empirically evaluate the effectiveness on two neural networks: AlexNet and ResNet-50 trained with the ImageNet-1k dataset while preserving the state-of-the-art test accuracy. Compared to the baseline of a previous study from a group of researchers at Facebook, our approach shows higher test accuracy on batch sizes that are larger than 16K. Using 2,048 Intel Xeon Platinum 8160 processors, we reduce the 100-epoch AlexNet training time from hours to 11 minutes. With 2,048 Intel Xeon Phi 7250 Processors, we reduce the 90-epoch ResNet-50 training time from hours to 20 minutes. Our implementation is open source and has been released in the Intel distribution of Caffe v1.0.7.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,199
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294 · cited 15×
“(2) Warmup Scheme (Goyal et al. 2017): If we use a large learning rate (η).”
BackgroundMethodResultInfluentialA high-performance, portable implementation of the MPI message passing interface standard
Gropp, W., Lusk, E., Doss, N., & Skjellum, A. (1996) · Parallel Computing · doi:10.1016/0167-8191(96)00024-5 · Cited by 2,222
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,110
Greenspan, H., van Ginneken, B., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2553401 · Cited by 1,700
Building high-level features using large scale unsupervised learning
Le, Q. V. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639343 · Cited by 1,695 · cited 1×
“parallelism can get the same solution as the single-machine case. Figure 2-(b) shows an example of using 4 machines to parallelize a 5-layer DNN. Model paralleism has been studied in (Catanzaro 2013; Le 2013). However, since the input size (e.g. size of an image) is relatively small, the matrix operations are not larg…”
BackgroundHogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
Niu, F., Recht, B., Re, C., & Wright, S. J. (2011) · arXiv · doi:10.48550/arxiv.1106.5730 · Cited by 1,211
One weird trick for parallelizing convolutional neural networks
Krizhevsky, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1404.5997 · Cited by 919
Optimization of Collective Communication Operations in MPICH
Thakur, R., Rabenseifner, R., & Gropp, W. (2005) · The International Journal of High Performance Computing Applications · doi:10.1177/1094342005051521 · Cited by 911
(2013) · Cited by 651
Revisiting Distributed Synchronous SGD
Pan, X., Chen, J., Monga, R., Bengio, S., & Józefowicz, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.05800 · Cited by 548
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., & Tang, P. T. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.04836 · Cited by 525
Seide, F., Fu, H., Droppo, J., Li, G., & Yu, D. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-274 · Cited by 443
FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters
Iandola, F. N., Moskewicz, M. W., Ashraf, K., & Keutzer, K. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.284 · Cited by 280
Self-citationExtremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
Akiba, T., Suzuki, S., & Fukuda, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.04325 · Cited by 260
TOP500 Supercomputer sites 11/2000
Meuer, H. W., Strohmaier, E., Dongarra, J. J., & Simon, H. D. (2000) · doi:10.2172/843058 · Cited by 220
Scaling SGD Batch Size to 32K for ImageNet Training.Preprint
You, Y., Gitman, I., & Ginsburg, B. (2017) · arXiv (Cornell University) · Cited by 212
Self-citationDistributed Deep Learning Using Synchronous Stochastic Gradient DescentPreprint
Das, D., Avancha, S., Mudigere, D., Vaidynathan, K., Sridharan, S., Kalamkar, D. D., Kaul, B., & Dubey, P. K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.06709 · Cited by 207
Optimization of Collective Reduction Operations
Rabenseifner, R. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24685-5_1 · Cited by 185
Asynchrony begets momentum, with an application to deep learning
Mitliagkas, I., Zhang, C., Hadjis, S., & Ré, C. (2016) · Allerton Conference on Communication, Control, and Computing · doi:10.1109/allerton.2016.7852343 · Cited by 156
How to scale distributed deep learning?
Jin, P. H., Yuan, Q., Iandola, F., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1611.04581 · Cited by 85
Self-citationOn parallelizability of stochastic gradient descent for speech DNNS
Seide, F. T. B., Fu, H., Droppo, J., Li, G., & Yu, D. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6853593 · Cited by 78
ImageNet Training in MinutesPreprint
You, Y., Zhang, Z., Hsieh, C., Demmel, J. W., & Keutzer, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.05011 · Cited by 38
Self-citationSequence Mining and Pattern Analysis in Drilling Reports with Deep Natural Language Processing
Hoffimann, J., Mao, Y., Wesley, A., & Taylor, A. (2018) · SPE Annual Technical Conference and Exhibition · doi:10.2118/191505-ms · Cited by 35
Cited by
Dota 2 with Large Scale Deep Reinforcement Learning
OpenAI, :, Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.06680 · Cited by 1,065
Efficient large-scale language model training on GPU clusters using megatron-LM
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. (2021) · Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1145/3458817.3476209 · Cited by 639
“[47] both demonstrate the use of High Performance Computing techniques to train highly-accurate ImageNet models in minutes.”
MethodLearning in the Frequency Domain
Xu, K., Qin, M., Sun, F., Wang, Y., Chen, Y.-K., & Ren, F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00181 · Cited by 563
Pneumonia Classification Using Deep Learning from Chest X-ray Images During COVID-19
Ibrahim, A. U., Ozsoz, M., Serte, S., Al-Turjman, F., & Yakoi, P. S. (2024) · Cognitive Computation · doi:10.1007/s12559-020-09787-5 · Cited by 419
Lookahead Optimizer: k steps forward, 1 step back
Zhang, M. R., Lucas, J., Hinton, G. E., & Ba, J. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.08610 · Cited by 345
Metadata
- Authors
- You, Y., Zhang, Z., Hsieh, C.-J., Demmel, J., & Keutzer, K.
- Venue
- Proceedings of the 47th International Conference on Parallel Processing
- Publisher
- ACM
- Published
- 2018-08-13
- Pages
- 1-10
- ISSN
- 0190-3918 (ISSN-L)
- OpenAlex
- W2962747323
- Cited by
- 384
- References
- 27
- FWCI
- 55.93
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.