The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
ImageNet Training in Minutes
Abstract
In this paper, we investigate large scale computers' capability of speeding up deep neural networks (DNN) training. Our approach is to use large batch size, powered by the Layer-wise Adaptive Rate Scaling (LARS) algorithm, for efficient usage of massive computing resources. Our approach is generic, as we empirically evaluate the effectiveness on two neural networks: AlexNet and ResNet-50 trained with the ImageNet-1k dataset while preserving the state-of-the-art test accuracy. Compared to the baseline of a previous study from a group of researchers at Facebook, our approach shows higher test accuracy on batch sizes that are larger than 16K. Using 2,048 Intel Xeon Platinum 8160 processors, we reduce the 100-epoch AlexNet training time from hours to 11 minutes. With 2,048 Intel Xeon Phi 7250 Processors, we reduce the 90-epoch ResNet-50 training time from hours to 20 minutes. Our implementation is open source and has been released in the Intel distribution of Caffe v1.0.7.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294 · cited 15×
“(2) Warmup Scheme (Goyal et al. 2017): If we use a large learning rate (η).”
BackgroundMethodResultInfluentialA high-performance, portable implementation of the MPI message passing interface standard
Gropp, W., Lusk, E., Doss, N., & Skjellum, A. (1996) · Parallel Computing · doi:10.1016/0167-8191(96)00024-5 · Cited by 2,222
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,110
Greenspan, H., van Ginneken, B., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2553401 · Cited by 1,700
Building high-level features using large scale unsupervised learning
Le, Q. V. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639343 · Cited by 1,695 · cited 1×
“parallelism can get the same solution as the single-machine case. Figure 2-(b) shows an example of using 4 machines to parallelize a 5-layer DNN. Model paralleism has been studied in (Catanzaro 2013; Le 2013). However, since the input size (e.g. size of an image) is relatively small, the matrix operations are not larg…”
BackgroundHogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
Niu, F., Recht, B., Re, C., & Wright, S. J. (2011) · arXiv · doi:10.48550/arxiv.1106.5730 · Cited by 1,211
Cited by
Dota 2 with Large Scale Deep Reinforcement Learning
OpenAI, :, Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.06680 · Cited by 1,065
Efficient large-scale language model training on GPU clusters using megatron-LM
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. (2021) · Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1145/3458817.3476209 · Cited by 639
“[47] both demonstrate the use of High Performance Computing techniques to train highly-accurate ImageNet models in minutes.”
MethodLearning in the Frequency Domain
Xu, K., Qin, M., Sun, F., Wang, Y., Chen, Y.-K., & Ren, F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00181 · Cited by 563
Pneumonia Classification Using Deep Learning from Chest X-ray Images During COVID-19
Ibrahim, A. U., Ozsoz, M., Serte, S., Al-Turjman, F., & Yakoi, P. S. (2024) · Cognitive Computation · doi:10.1007/s12559-020-09787-5 · Cited by 419
Lookahead Optimizer: k steps forward, 1 step back
Zhang, M. R., Lucas, J., Hinton, G. E., & Ba, J. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.08610 · Cited by 345
MegDet: A Large Mini-Batch Object Detector
Peng, C., Xiao, T., Li, Z., Jiang, Y., Zhang, X., Jia, K., Yu, G., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00647 · Cited by 311
Generative Adversarial Networks for face generation: A survey
Kammoun, A., Slama, R., Tabia, H., Ouni, T., & Abid, M. (2022) · ACM Computing Surveys · doi:10.1145/1122445.1122456 · Cited by 311
Jia, X., Song, S., He, W., Wang, Y., Rong, H., Zhou, F., Xie, L., Guo, Z., Yang, Y., Yu, L.-W., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.11205 · Cited by 281
The Architectural Implications of Facebook's DNN-Based Personalized Recommendation
Gupta, U., Wu, C.-J., Wang, X., Naumov, M., Reagen, B., Brooks, D., Cottel, B., Hazelwood, K., Hempstead, M., Jia, B., et al. (2020) · 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca47549.2020.00047 · Cited by 273
Single chip photonic deep neural network with accelerated training
Bandyopadhyay, S., Sludds, A., Krastanov, S., Hamerly, R., Harris, N., Bunandar, D., Streshinsky, M., Hochberg, M., & Englund, D. (2022) · arXiv · doi:10.48550/arxiv.2208.01623 · Cited by 265
Industrial Internet of Things Intelligence Empowering Smart Manufacturing: A Literature Review
Hu, Y., Jia, Q., Yao, Y., Lee, Y., Lee, M., Wang, C., Zhou, X., Xie, R., & Yu, F. R. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3367692 · Cited by 256
Mazzia, V., Khaliq, A., Salvetti, F., & Chiaberge, M. (2020) · IEEE Access · doi:10.1109/access.2020.2964608 · Cited by 212
A survey of techniques for optimizing deep learning on gpus
Mittal, S., & Vaishay, S. (2019) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2019.101635 · Cited by 197
Multimodal datasets: misogyny, pornography, and malignant stereotypes
Birhane, A., Prabhu, V. U., & Kahembwe, E. (2021) · arXiv · doi:10.48550/arxiv.2110.01963 · Cited by 160
Toward Communication-Efficient Federated Learning in the Internet of Things With Edge Computing
Sun, H., Li, S., Yu, F. R., Qi, Q., Wang, J., & Liao, J. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2994596 · Cited by 95
The Impact of GPU DVFS on the Energy and Performance of Deep Learning: an Empirical Study
Tang, Z., Wang, Y., Wang, Q., & Chu, X. (2019) · Proceedings of the Tenth ACM International Conference on Future Energy Systems · doi:10.1145/3307772.3328315 · Cited by 88
“While most of the previous work addressed the model accuracy and training performance [13, 15, 47], the energy consumption of those high-throughput GPU machines is usually overlooked.”
ResultData Movement Is All You Need: A Case Study on Optimizing Transformers
Ivanov, A., Dryden, N., Ben-Nun, T., Li, S., & Hoefler, T. (2020) · arXiv · doi:10.48550/arxiv.2007.00072 · Cited by 78
Taming unbalanced training workloads in deep learning with partial collective operations
Li, S., Ben-Nun, T., Girolamo, S. D., Alistarh, D., & Hoefler, T. (2020) · Proceedings of the 25th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3332466.3374528 · Cited by 57
Rasch, M. J., Gokmen, T., Rigotti, M., & Haensch, W. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.00753 · Cited by 14
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., & Hsieh, C.-J. (2019) · arXiv · doi:10.48550/arxiv.1904.00962
Self-citation
Metadata
- Authors
- You, Y., Zhang, Z., Hsieh, C.-J., Demmel, J., & Keutzer, K.
- Venue
- Proceedings of the 47th International Conference on Parallel Processing
- Publisher
- ACM
- Published
- 2018-08-13
- Pages
- 1-10
- ISSN
- 0190-3918 (ISSN-L)
- OpenAlex
- W2962747323
- Cited by
- 384
- References
- 27
- FWCI
- 55.93
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.