Taming unbalanced training workloads in deep learning with partial collective operations
Open accessThe DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index.
Abstract
Load imbalance pervasively exists in distributed deep learning training systems, either caused by the inherent imbalance in learned tasks or by the system itself. Traditional synchronous Stochastic Gradient Descent (SGD) achieves good accuracy for a wide variety of tasks, but relies on global synchronization to accumulate the gradients at every training step. In this paper, we propose eager-SGD, which relaxes the global synchronization for decentralized accumulation. To implement eager-SGD, we propose to use two partial collectives: solo and majority. With solo allreduce, the faster processes contribute their gradients eagerly without waiting for the slower processes, whereas with majority allreduce, at least half of the participants must contribute gradients before continuing, all without using a central parameter server. We theoretically prove the convergence of the algorithms and describe the partial collectives in detail. Experiments are conducted on a variety of neural networks and datasets. The results on load-imbalanced environments show that eager-SGD achieves 2.64 X speedup (ResNet-50 on ImageNet) over the asynchronous centralized SGD, and achieves 1.29 X speedup (ResNet-50 on ImageNet) and 1.27X speedup (LSTM on UCF101) over the state-of-the-art synchronous decentralized SGDs, without losing accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 3×
“Residual Network (ResNet) [21] is widely used in computer vision tasks.”
MethodVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“Transformers [57] are sequence-to-sequence models that translate a sequence of words from one language to another.”
BackgroundVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“Early convolutional networks demonstrated groundbreaking successes in computer vision, ranging from image classification to object detection [29, 49].”
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“Long short-term memory (LSTM) [23] is a type of unit cell in Recurrent Neural Networks (RNN).”
BackgroundLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“deoclassification,naturallanguageprocessingfor machinetranslation,questionanswering,textcomprehension,andsynthetictextgeneration.Thelattermodels containmorethan1.5billionparametersandtakeweeks totrain[14,42].Otherdemandingneuralnetworksare trainedonthelargestsupercomputerstoachievescientificbreakthroughs[33,38].Furthe…”
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“4 shows the runtime distribution over the sampled batches for 5 epochs of training for the classic ResNet-50 model [21] on ImageNet [14], on a standard Google Cloud instance (n1-standard-16 with 2x Nvidia V100 GPUs).”
MethodGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×
“The gradient of those functions can be computed using the backpropagation algorithm [35], processing the nodes in the DAG in a reverse topological order.”
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643 · cited 1×
“Supervised deep neural network training typically involves first-order optimization in the form of Stochastic Gradient Descent (SGD) [48].”
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575 · cited 4×
“2a shows the video length distribution (the number of frames) over all 9,537 videos in the training dataset of UCF101 [53].”
BackgroundMethodInfluentialLarge Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
Improved regularization of convolutional neural networks with cutout
DeVries, T., & Taylor, G. W. (2017) · arXiv · doi:10.48550/arxiv.1708.04552 · Cited by 2,776 · cited 2×
“In fact, data augmentations such as Cutout [16] and Dropout [6] introduce random errors and omissions into the training process to improve generalization properties.”
ImageNet: A large-scale hierarchical image database
Deng, J. H., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714
Proceedings of the 13th International Conference on Neural Information Processing Systems
Leen, T. K., Dietterich, T. G., & Tresp, V. (2000) · Cited by 2,190
Beyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299101 · Cited by 2,062 · cited 1×
“In video classification tasks, LSTMs are used [7, 18, 60] to process a sequence of frames for a video as input (optionally following convolutional neural networks that preprocess the images to features), and output a probability distribution over a set of classes.”
MethodHorovod: fast and easy distributed deep learning in TensorFlow
Sergeev, A., & Del Balso, M. (2018) · arXiv · doi:10.48550/arxiv.1802.05799 · Cited by 1,620 · cited 4×
“Horovod [48] is one such implementation over the TensorFlow framework, which also fuses several”
InfluentialLong-Term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2599174 · Cited by 1,547
Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
Niu, F., Recht, B., Re, C., & Wright, S. J. (2011) · arXiv · doi:10.48550/arxiv.1106.5730 · Cited by 1,211 · cited 1×
“This enables the PS to asynchronously update the global view [46], or require a fraction of learners to send gradients before progressing to the next step [22].”
BackgroundScaling distributed machine learning with the parameter server
Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed, A. A., Josifovski, V., Long, J. E., Shekita, E. J., & Su, B.-Y. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685095 · Cited by 1,021
Scaling Distributed Machine Learning with the Parameter Server
Li, M. (2014) · Proceedings of the 2014 International Conference on Big Data Science and Computing · doi:10.1145/2640087.2644155 · Cited by 979
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. (2014) · doi:10.21236/ada623249 · Cited by 968
Optimization of Collective Communication Operations in MPICH
Thakur, R., Rabenseifner, R., & Gropp, W. (2005) · The International Journal of High Performance Computing Applications · doi:10.1177/1094342005051521 · Cited by 911 · cited 1×
“Collective communication Several algorithms can be used to implement allreduce operations, and the optimal algorithm depends on network topology, number of processes, and message size [53].”
Runtime measurements in the cloud: observing, analyzing, and reducing variance
Schad, J., Dittrich, J., & Quiané-Ruiz, J.-A. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1920902 · Cited by 670 · cited 3×
“finishthecomputationlaterthanothers, allprocesseswillwaitforthelastoneattheblocking allreduce function. Load imbalance can be caused by thesystemitself,forexample,whentrainingonmultitenantcloudsystems[30,31,46]orbysystemornetwork noise[26,27]inhigh-performancemachines.Asecond, andmoreprominentcauseofimbalanceisinheren…”
BackgroundInfluentialPerformance Analysis of High Performance Computing Applications on the Amazon Web Services Cloud
Jackson, K. R., Ramakrishnan, L., Muriki, K., Canon, S., Cholia, S., Shalf, J., Wasserman, H. J., & Wright, N. J. (2010) · 2010 IEEE Second International Conference on Cloud Computing Technology and Science · doi:10.1109/cloudcom.2010.69 · Cited by 596 · cited 3×
“Load imbalance can be caused by the system itself, for example, when training on multi-tenant cloud systems [31, 32, 49] or by system or network noise [27, 28] in high-performance machines.”
BackgroundDemystifying Parallel and Distributed Deep Learning: An In-depth Concurrency Analysis
Ben-Nun, T., & Hoefler, T. (2020) · ACM Computing Surveys · doi:10.1145/3320060 · Cited by 577 · cited 3×
“The communication aspects of deep learning have been investigated in many different contexts [47, 51], see the survey for an overview [10].”
InfluentialSelf-citationProject Adam: building an efficient and scalable deep learning training system
Chilimbi, T. M., Suzue, Y., Apacible, J., & Kalyanaraman, K. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685094 · Cited by 550
The Convergence of Sparsified Gradient MethodsPreprint
Alistarh, D., Hoefler, T., Johansson, M., Khirirat, S., Konstantinov, N., & Renggli, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10505 · Cited by 530 · cited 1×
“Several packages take advantage of this robustness and employ three techniques in tandem: (1) communicated weights are quantized to more compact number representations [50, 54], (2) only the most significant weights are sent during each allreduce [2, 47], and (3) updates are only sent to limited (random) neighborhoods…”
MethodSelf-citationScalable distributed DNN training using commodity GPU cloud computing
Strom, N. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-354 · Cited by 521 · cited 1×
“Several packages take advantage of this robustness and employ three techniques in tandem: (1) communicated weights are quantized to more compact number representations [50, 54], (2) only the most significant weights are sent during each allreduce [2, 47], and (3) updates are only sent to limited (random) neighborhoods…”
MethodMore Effective Distributed ML via a Stale Synchronous Parallel Parameter Server
Ho, Q., Cipar, J., Cui, H., Kim, J. K., Lee, S., Gibbons, P. B., Gibson, G. A., Ganger, G. R., & Xing, E. P. (2013) · PubMed · doi:10.1184/r1/6475898 · Cited by 506
Seide, F., Fu, H., Droppo, J., Li, G., & Yu, D. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-274 · Cited by 443 · cited 1×
“Several packages take advantage of this robustness and employ three techniques in tandem: (1) communicated weights are quantized to more compact number representations [47, 51], (2) only the most significant weights are sent during each allreduce [2, 44], and (3) updates are only sent to limited (random) neighborhoods…”
MethodAsynchronous Decentralized Parallel Stochastic Gradient DescentPreprint
Lian, X., Zhang, W., Zhang, C., & Liu, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.06952 · Cited by 436 · cited 3×
“, forming a ring [40] or a general graph [58]; or on a random subset using Gossip Algorithms [12, 34].”
MethodAdaptive dropout for training deep neural networks
Ba, J. L., & Frey, B. J. (2013) · neural information processing systems · Cited by 416
Bandwidth optimal all-reduce algorithms for clusters of workstations
Patarasuk, P., & Yuan, X. (2009) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2008.09.002 · Cited by 409 · cited 1×
“Yet, the optimal scaling of an allreduce of size S is at best O (log P + S) in P processes [25, 40, 44].”
BackgroundLian, X., Zhang, C., Zhang, H., Hsieh, C., Zhang, W., & Liu, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09056 · Cited by 395 · cited 3×
“Then, we use the different neural networks summarized in Table 1 to compare our eager-SGD with the state-of-the-art synchSGD implementations (Horovod [51] and Deep500 [9]), the asynchronous centralized SGD [1], and the gossip-based SGDs [4, 39], under simulated and real workload imbalance environments.”
BackgroundMethodYou, Y., Zhang, Z., Hsieh, C.-J., Demmel, J., & Keutzer, K. (2018) · Proceedings of the 47th International Conference on Parallel Processing · doi:10.1145/3225058.3225069 · Cited by 384 · cited 1×
“However, large batch sizes commonly need carefully-tuned learning rate schedules to achieve good accuracy [56], which is out of scope.”
On the Performance Variability of Production Cloud Services
Iosup, A., Yigitbasi, N., & Epema, D. (2011) · 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing · doi:10.1109/ccgrid.2011.22 · Cited by 328 · cited 3×
“Load imbalance can be caused by the system itself, for example, when training on multi-tenant cloud systems [31, 32, 49] or by system or network noise [27, 28] in high-performance machines.”
BackgroundA Stochastic Approximation Method
Robbins, H., & Monro, S. (1985) · Herbert Robbins Selected Papers · doi:10.1007/978-1-4612-5110-1_9 · Cited by 284
Exascale Deep Learning for Climate Analytics
Kurth, T., Treichler, S., Romero, J., Mudigonda, M., Luehr, N., Phillips, E., Mahesh, A., Matheson, M., Deslippe, J., Fatica, M., et al. (2018) · SC18: International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc.2018.00054 · Cited by 278
Characterizing the Influence of System Noise on Large-Scale Applications by Simulation
Hoefler, T., Schneider, T., & Lumsdaine, A. (2010) · 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc.2010.12 · Cited by 220
Self-citationDemystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
Ben‐Nun, T., & Hoefler, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09941 · Cited by 203 · cited 3×
“The communication aspects of deep learning have been investigated in many different contexts [47, 51], see the survey for an overview [10].”
InfluentialSelf-citationPriority-based Parameter Propagation for Distributed DNN TrainingPreprint
Jayarajan, A., Wei, J., Gibson, G. A., Fedorova, A., & Pekhimenko, G. G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.03960 · Cited by 199 · cited 1×
“For synchronous Parameter Server SGD, communication granularity and scheduling optimizations [33] are studied to better overlap communication and computation.”
BackgroundImplementation and performance analysis of non-blocking collective operations for MPI
Hoefler, T., Lumsdaine, A., & Rehm, W. (2007) · Proceedings of the 2007 ACM/IEEE conference on Supercomputing · doi:10.1145/1362622.1362692 · Cited by 193 · cited 1×
“The allreduce operation is not atomic and it can be split into layer-wise reductions, which can easily be overlapped with the layer computation using non-blocking collectives [4, 24].”
BackgroundSelf-citationOptimization of Collective Reduction Operations
Rabenseifner, R. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24685-5_1 · Cited by 185 · cited 2×
“For large message sizes and large number of processes, practical implementations employ the ring-allreduce [19] or the Rabenseifner’s Algorithm [44].”
Staleness-aware Async-SGD for Distributed Deep Learning
Zhang, W., Gupta, S., Lian, X., & Liu, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05950 · Cited by 171 · cited 1×
“Deep learning Parameter Server SGD implementations use synchronous [19, 36], asynchronous [10, 12], stale- [21, 59], and approximate-synchronous [28] SGD, where the latter two limit the age and insignificance of the gradients, respectively.”
Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study
Gupta, S., Zhang, W., & Wang, F. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0028 · Cited by 139
CosmoFlow: Using Deep Learning to Learn the Universe at Scale
Mathuriya, A., Bard, D., Mendygral, P., Meadows, L., Arnemann, J., Shao, L., He, S., Karna, T., Moise, D., Pennycook, S. J., et al. (2018) · SC18: International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc.2018.00068 · Cited by 132
S-Caffe: Co-designing MPI Runtimes and Caffe for Scalable Deep Learning on Modern GPU Clusters
Awan, A. A., Hamidouche, K., Hashmi, J. M., & Panda, D. K. (2017) · Proceedings of the 22nd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3018743.3018769 · Cited by 128 · cited 2×
“The allreduce operation is not atomic and it can be split into layer-wise reductions, which can easily be overlapped with the layer computation using non-blocking collectives [5, 25].”
BackgroundMethodDelving Deeper into Convolutional Networks for Learning Video Representations
Ballas, N., Yao, L., Pal, C., & Courville, A. C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06432 · Cited by 126 · cited 1×
“In video classification tasks, LSTMs are used [7, 18, 60] to process a sequence of frames for a video as input (optionally following convolutional neural networks that preprocess the images to features), and output a probability distribution over a set of classes.”
MethodStochastic Gradient Push for Distributed Deep Learning
Assran, M., Loizou, N., Ballas, N., & Rabbat, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.10792 · Cited by 120 · cited 5×
“1 to compare our eager-SGD with the state-of-the-art synch-SGD implementations (Horovod [51] and Deep500 [9]), the asynchronous centralized SGD [1], and the gossip-based SGDs [4, 39], under simulated and real workload imbalance environments.”
InfluentialHow to scale distributed deep learning?
Jin, P. H., Yuan, Q., Iandola, F., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1611.04581 · Cited by 85 · cited 1×
“, forming a ring [40] or a general graph [58]; or on a random subset using Gossip Algorithms [12, 34].”
MethodGossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
Daily, J., Vishnu, A., Siegel, C., Warfel, T., & Amatya, V. (2018) · arXiv · doi:10.48550/arxiv.1803.05880 · Cited by 80 · cited 1×
“, forming a ring [40] or a general graph [58]; or on a random subset using Gossip Algorithms [12, 34].”
MethodA Modular Benchmarking Infrastructure for High-Performance and Reproducible Deep Learning
Ben-Nun, T., Besta, M., Huber, S., Ziogas, A. N., Peter, D., & Hoefler, T. (2019) · 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2019.00018 · Cited by 76 · cited 3×
“Then, we use the different neural networks summarized in Table 1 to compare our eager-SGD with the state-of-the-art synchSGD implementations (Horovod [51] and Deep500 [9]), the asynchronous centralized SGD [1], and the gossip-based SGDs [4, 39], under simulated and real workload imbalance environments.”
MethodSelf-citationsPIN: High-performance streaming Processing In the Network
Hoefler, T., Girolamo, S. D., Taranov, K., Grant, R. E., & Brightwell, R. (2017) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3126908.3126970 · Cited by 74 · cited 1×
“This limit can be removed by implementing the schedule execution with the sPIN programming model [23], which allows to execute user-defined code on the NIC.”
MethodSelf-citationModel Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study
Gupta, S., Zhang, W., & Wang, F. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/681 · Cited by 53 · cited 1×
“Deep learning Parameter Server SGD implementations use synchronous [19, 36], asynchronous [10, 12], stale- [21, 59], and approximate-synchronous [28] SGD, where the latter two limit the age and insignificance of the gradients, respectively.”
Energy, Memory, and Runtime Tradeoffs for Implementing Collective Communication Operations
Hoefler, T., & Moor, D. (2014) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000093896 · Cited by 32
Self-citationSparCML: High-Performance Sparse Communication for Machine LearningPreprint
Renggli, C., Ashkboos, S., Aghagolzadeh, M., Alistarh, D., & Hoefler, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.08021 · Cited by 31
Self-citationSandia National Labs., A., Dirk, S. M., & (NNSA), U. N. N. S. A. (2018) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/1761886 · Cited by 28
Exploiting Offload Enabled Network Interfaces
Girolamo, S. D., Jolivet, P., Underwood, K. D., & Hoefler, T. (2015) · IEEE Symposium on High-Performance Interconnects (HOTI) · doi:10.1109/hoti.2015.21 · Cited by 18 · cited 2×
“[17] show how solo collectives can be offloaded to Portals 4 [8] NICs by using triggered operations.”
BackgroundSelf-citationLighter-communication distributed machine learning via Sufficient Factor Broadcasting
Xie, P., Kim, J. K., Zhou, Y. X., Ho, Q., Kumar, A., Yu, Y., & Xing, E. P. (2016) · Uncertainty in Artificial Intelligence · Cited by 17
Model Accuracy and Runtime Tradeoff in Distributed Deep LearningPreprint
Gupta, S., Zhang, W., & Milthorpe, J. (2015) · arXiv (Cornell University) · Cited by 16
THE EFFECT OF NETWORK NOISE ON LARGE-SCALE COLLECTIVE COMMUNICATIONS
Hoefler, T., Schneider, T., & Lumsdaine, A. (2009) · Parallel Processing Letters · doi:10.1142/s0129626409000420 · Cited by 12
Self-citationsPIN: High-performance streaming Processing in the Network.
Hoefler, T. (2018) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · Cited by 12
Self-citationLong-term Recurrent Convolutional Networks for Visual Recognition and DescriptionPreprint
Donahue, J., Hendricks, L. A., Rohrbach, M., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.4389 · Cited by 12 · cited 1×
“In video classification tasks, LSTMs are used [6, 17, 57] to process a sequence of frames for a video as input (optionally following convolutional neural networks that preprocess the images to features), and output a probability distribution over a set of classes.”
MethodModel Accuracy and Runtime Tradeoff in Distributed Deep Learning:A Systematic StudyPreprint
Gupta, S., Zhang, W., & Wang, F. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.04210 · Cited by 9 · cited 1×
“Deep learning Parameter Server SGD implementations use synchronous [19, 36], asynchronous [10, 12], stale- [21, 59], and approximate-synchronous [28] SGD, where the latter two limit the age and insignificance of the gradients, respectively.”
Research works printed but not linked: 4
- [19] Andrew Gibiansky. 2017. Bringing HPC techniques to deep learning.(2017). URL http://research. baidu. com/bringing-hpc-techniquesdeep-learning (2017). -
- [29] Kevin Hsieh, Aaron Harlap, Nandita Vijaykumar, Dimitris Konomis, Gregory R. Ganger, Phillip B. Gibbons, and Onur Mutlu. 2017. Gaia: Geo-distributed Machine Learning Approaching LAN Speeds. In Proceedings of the 14th USENIX Conference on Networked Systems Design and Implementation (NSDI'17) . USENIX Association, Berkeley, CA, USA, 629-647. http://dl.acm.org/citation.cfm?id=3154630.3154682 -
- [42] Message Passing Interface Forum. 2015. MPI: A Message-Passing Interface Standard Version 3.1. -
- [61] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. 2015. Beyond short snippets: Deep networks for video classification. In Proceedings of the IEEE conference on computer vision and pattern recognition . 4694-4702. - · DOI · Paper page
Cited by
Energy efficiency in cloud computing data centers: a survey on software technologies
Katal, A., Dahiya, S., & Choudhury, T. (2023) · Cluster Computing · doi:10.1007/s10586-022-03713-0 · Cited by 516
“…techniques there are various other supervised learning techniques like Support Vector Machine (SVM) [103], Stochastic Gradient Descent (SGD) [104], Logistic Regression (LR) [105], Random Forest (RF) [106], Multi-Layer Perceptron (MLP) [107], Backpropagation neural network [108] were used.”
MethodAn In-Depth Analysis of the Slingshot Interconnect
Sensi, D. D., Girolamo, S. D., McMahon, K. H., Roweth, D., & Hoefler, T. (2020) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41405.2020.00039 · Cited by 154
“Resnet-proxy This is a ML/AI proxy application [41], reproducing the communication phases of a Deep500 benchmark [42] Residual Neural Network (resnet).”
MethodSelf-citationChimera: efficiently training large-scale neural networks with bidirectional pipelines
Li, S., & Hoefler, T. (2021) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3458817.3476145 · Cited by 223
“To scale up the training process to parallel machines, data parallelism [20, 33, 49, 60] is the common method, in which the minibatch is partitioned among P workers and each worker maintains a copy of the entire model.”
MethodSelf-citationFasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models
He, J., Zhai, J., Antunes, T., Wang, H., Luo, F., Shi, S., & Li, Q. (2022) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3503221.3508418 · Cited by 258
“Asynchronous partial model updating methods [15, 16, 18, 32] are introduced to speed up convergence in heterogeneous environment for data parallelism.”
MethodParallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis
Besta, M., & Hoefler, T. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3303431 · Cited by 106
“Thismaycause load imbalance ), similarly to, e.g., videos [16].”
Self-citation
Metadata
- Authors
- Li, S., Ben-Nun, T., Girolamo, S. D., Alistarh, D., & Hoefler, T.
- Affiliations
- ETH Zurich (CH)
- ETH Zurich
- Institute of Science and Technology Austria (AT)
- Venue
- Proceedings of the 25th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming
- Publisher
- ACM
- Published
- 2020-02-19
- Pages
- 45-61
- OpenAlex
- W2967558351
- Cited by
- 57
- References
- 67
- Citations by year
- 2026: 2 · 2025: 10 · 2024: 7 · 2023: 8 · 2022: 11 · 2021: 11 · 2020: 8
- FWCI
- 3.63
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- European Research Council (ERC) under the European Union?s Horizon 2020 programme, grant agreement DAPP, (No. 678880)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- load imbalance, distributed deep learning training, AllReduce, asynchronous SGD, convergence analysis, speedup
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.