The DOI matches its OpenAlex record; 25 of the 25 references it cites were found in a registry or an index.
Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
Open accessOverview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
(2016) · Cited by 11,679
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294 · cited 1×
“[11] train the ResNet-50 model with a large mini-batch size of 8K over 256 Tesla GPUs and finish the training process within one hour.”
Method(1992) · Cited by 1,405
Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al. (2017) · arXiv · doi:10.48550/arxiv.1710.03740 · Cited by 1,179
Optimization of Collective Communication Operations in MPICH
Thakur, R., Rabenseifner, R., & Gropp, W. (2005) · The International Journal of High Performance Computing Applications · doi:10.1177/1094342005051521 · Cited by 911
Don't Decay the Learning Rate, Increase the Batch Size
Smith, S. L., Kindermans, P.-J., Ying, C., & Le, Q. V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00489 · Cited by 559
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., & Tang, P. T. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.04836 · Cited by 525
Horovod: fast and easy distributed deep learning in TensorFlow
Sergeev, A., & Balso, M. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.05799 · Cited by 472
Large Batch Training of Convolutional Networks
You, Y., Gitman, I., & Ginsburg, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.03888 · Cited by 463
Training deep neural networks with low precision multiplications
Courbariaux, M., Bengio, Y., & David, J.-P. (2014) · arXiv · doi:10.48550/arxiv.1412.7024 · Cited by 407
You, Y., Zhang, Z., Hsieh, C.-J., Demmel, J., & Keutzer, K. (2018) · Proceedings of the 47th International Conference on Parallel Processing · doi:10.1145/3225058.3225069 · Cited by 384
Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
Akiba, T., Suzuki, S., & Fukuda, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.04325 · Cited by 260
Scaling SGD Batch Size to 32K for ImageNet Training.Preprint
You, Y., Gitman, I., & Ginsburg, B. (2017) · arXiv (Cornell University) · Cited by 212
AdaBatch: Adaptive Batch Sizes for Training Deep Neural NetworksPreprint
Devarakonda, A., Naumov, M., & Garland, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.02029 · Cited by 177
Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
Shi, S., Wang, Q., & Chu, X. (2018) · IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech) · doi:10.1109/dasc/picom/datacom/cyberscitec.2018.000-4 · Cited by 146
Self-citationInterprocessor collective communication library (InterCom)
Barnett, M., Shuler, L., Geijn, R. A., Gupta, S., Payne, D. G., & Watts, J. (2002) · IEEE Scalable High Performance Computing Conference (SHPCC) · doi:10.1109/shpcc.1994.296665 · Cited by 142
High-Accuracy Low-Precision TrainingPreprint
De, C., Leszczynski, M., Zhang, J., Marzoev, A., Aberger, C. R., Olukotun, K., & Ré, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.03383 · Cited by 111
Ako: Decentralised Deep Learning with Partial Gradient Exchange
Watcharapichat, P., Morales, V. L., Fernandez, R. C., & Pietzuch, P. R. (2016) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/2987550.2987586 · Cited by 91
Codreanu, V., Podareanu, D., & Saletore, V. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.04291 · Cited by 49
A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning
Shi, S., Wang, Q., Chu, X., & Li, B. (2018) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/padsw.2018.8644932 · Cited by 27
Self-citation
Cited by
Bag of Tricks for Image Classification with Convolutional Neural Networks
He, T., Zhang, Z., Zhang, H., Zhang, Z., Xie, J., & Li, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00065 · Cited by 1,680
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
Deng, B. L., Li, G., Han, S., Shi, L., & Xie, Y. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.2976475 · Cited by 996
“This is the reason that most quantization methods remain the master weight and its update in high precision [177]–[179], [193].”
BackgroundPipeDream: generalized pipeline parallelism for DNN training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., Gibbons, P. B., & Zaharia, M. (2019) · Proceedings of the 27th ACM Symposium on Operating Systems Principles · doi:10.1145/3341301.3359646 · Cited by 818
Flower: A Friendly Federated Learning Research Framework
Beutel, D. J., Topal, T., Mathur, A., Qiu, X., Fernandez-Marques, J., Gao, Y., Sani, L., Li, K. H., Parcollet, T., de Gusmão, P. P. B., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14390 · Cited by 622
“The optimization of distributed training with and without federated concepts have been covered from many angles [15, 16, 17, 18, 19].”
BackgroundDemystifying Parallel and Distributed Deep Learning: An In-depth Concurrency Analysis
Ben-Nun, T., & Hoefler, T. (2020) · ACM Computing Surveys · doi:10.1145/3320060 · Cited by 577
Metadata
- Authors
- Jia, X., Song, S., He, W., Wang, Y., Rong, H., Zhou, F., Xie, L., Guo, Z., Yang, Y., Yu, L.-W., Chen, T., Hu, G., Shi, S., & Chu, X.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2018-07-30
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1807.11205
- OpenAlex
- W2884711234
- Cited by
- 281
- References
- 25
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.