The DOI matches its Crossref record; 255 of the 255 references it cites were found in a registry or an index.
Demystifying Parallel and Distributed Deep Learning: An In-depth Concurrency Analysis
Abstract
Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this survey, we describe the problem from a theoretical perspective, followed by approaches for its parallelization. We present trends in DNN architectures and the resulting implications on parallelization strategies. We then review and model the different types of concurrency in DNNs: from the single operator, through parallelism in network inference and training, to distributed deep learning. We discuss asynchronous stochastic optimization, distributed system architectures, communication schemes, and neural architecture search. Based on those approaches, we extrapolate potential directions for parallelism in deep learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–255 of 255
Optimizing Memory Efficiency for Deep Convolutional Neural Networks on GPUs
Li, C., Yang, Y., Feng, M., Chakradhar, S., & Zhou, H. (2016) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc.2016.53 · Cited by 109
Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
De, C., Zhang, C., Olukotun, K., & Ré, C. (2015) · PubMed · doi:10.48550/arxiv.1506.06438 · Cited by 108
DeepRebirth: Accelerating Deep Neural Network Execution on Mobile Devices
Li, D., Wang, X., & Kong, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11876 · Cited by 101
Performance Modeling and Scalability Optimization of Distributed Deep Learning Systems
Yan, F., Ruwase, O., He, Y., & Chilimbi, T. M. (2015) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2783258.2783270 · Cited by 100
Large-batch training for LSTM and beyond
You, Y., Hseu, J., Ying, C., Demmel, J. W., Keutzer, K., & Hsieh, C. (2019) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3295500.3356137 · Cited by 96
Asynchronous gossip algorithms for stochastic optimization
Ram, S. S., Nedić, A., & Veeravalli, V. V. (2009) · International Conference on Game Theory for Networks · doi:10.1109/gamenets.2009.5137386 · Cited by 94
YellowFin and the Art of Momentum TuningPreprint
Zhang, J., & Mitliagkas, I. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.03471 · Cited by 93
Practical Network Blocks Design with Q-Learning.Preprint
Zhong, Z., Yan, J., & Liu, C. (2017) · arXiv (Cornell University) · Cited by 90
Hyper-parameter selection in deep neural networks using parallel particle swarm optimization
Lorenzo, P. R., Nalepa, J., Ramos, L. S., & Ranilla, J. (2017) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3067695.3084211 · Cited by 87
Integrated Model, Batch, and Domain Parallelism in Training Neural Networks
Gholami, A., Azad, A., Jin, P., Keutzer, K., & Buluç, A. (2018) · ACM Symposium on Parallelism in Algorithms and Architectures (SPAA) · doi:10.1145/3210377.3210394 · Cited by 87
How to scale distributed deep learning?
Jin, P. H., Yuan, Q., Iandola, F., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1611.04581 · Cited by 85
GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network TrainingPreprint
Paine, T., Jin, H., Yang, J., Lin, Z., & Huang, T. S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6186 · Cited by 84
Deep learning at 15PF: supervised and semi-supervised classification for scientific data
Kurth, T., Zhang, J., Nadathur, S., Racah, E., Mitliagkas, I., Patwary, M. M. A., Malas, T. B., Sundaram, N., Bhimji, W., Smorkalov, M. E., et al. (2017) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3126908.3126916 · Cited by 82
GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
Daily, J., Vishnu, A., Siegel, C., Warfel, T., & Amatya, V. (2018) · arXiv · doi:10.48550/arxiv.1803.05880 · Cited by 80
On parallelizability of stochastic gradient descent for speech DNNS
Seide, F. T. B., Fu, H., Droppo, J., Li, G., & Yu, D. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6853593 · Cited by 78
LBANN: livermore big artificial neural network HPC toolkit
Essen, B. V., Kim, H., Pearce, R., Boakye, K., & Chen, B. Y. (2015) · doi:10.1145/2834892.2834897 · Cited by 77
Optimizing Performance of Recurrent Neural Networks on GPUsPreprint
Appleyard, J., Kočiský, T., & Blunsom, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.01946 · Cited by 77
Scaling deep learning on GPU and knights landing clusters
You, Y., Buluç, A., & Demmel, J. W. (2017) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3126908.3126912 · Cited by 75
Towards Pervasive and User Satisfactory CNN across GPU Microarchitectures
Song, M., Hu, Y., Chen, H., & Li, T. (2017) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2017.52 · Cited by 74
Persistent RNNs: stashing recurrent weights on-chip
Diamos, G., Sengupta, S., Catanzaro, B., Chrzanowski, M., Coates, A., Elsen, E., Engel, J., Hannun, A., & Satheesh, S. (2016) · International Conference on Machine Learning · Cited by 72
Without-Replacement Sampling for Stochastic Gradient Methods
Shamir, O. (2016) · neural information processing systems · Cited by 72
Distributed Second-Order Optimization using Kronecker-Factored Approximations
Ba, J. L., Grosse, R., & Martens, J. (2017) · International Conference on Learning Representations · Cited by 66
Implementing Neural Network Models on Parallel Computers
Forrest, B. M., Roweth, D., Stroud, N., Wallace, D. J., & Wilson, G. (1987) · The Computer Journal · doi:10.1093/comjnl/30.5.413 · Cited by 61
Omnivore: An Optimizer for Multi-device Deep Learning on CPUs and GPUsPreprint
Hadjis, S., Zhang, C., Mitliagkas, I., Iter, D., & Ré, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04487 · Cited by 61
ImageNet/ResNet-50 Training in 224 Seconds.Preprint
Mikami, H., Suganuma, H., U‐Chupala, P., Tanaka, Y., & Kageyama, Y. (2018) · arXiv (Cornell University) · Cited by 58
Large-Scale Deep Belief Nets With MapReduce
Zhang, K., & Chen, X. (2014) · IEEE Access · doi:10.1109/access.2014.2319813 · Cited by 54
Image Classification Using Generative Neuro Evolution for Deep Learning
Verbancsics, P., & Harguess, J. D. (2015) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2015.71 · Cited by 52
Ben‐Nun, T., Levy, E., Barak, A., & Rubin, E. (2015) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/2807591.2807611 · Cited by 52
Self-citationLatte: a language, compiler, and runtime for elegant and efficient deep neural networks
Truong, L., Barik, R., Totoni, E., Liu, H., Markley, C., Fox, A., & Shpeisman, T. (2016) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2908080.2908105 · Cited by 51
Evolving Deep Networks Using HPC
Young, S. R., Rose, D. C., Johnston, T., Heller, W. T., Karnowski, T. P., Potok, T. E., Patton, R. M., Perdue, G., & Miller, J. (2017) · doi:10.1145/3146347.3146355 · Cited by 46
Improving Strong-Scaling of CNN Training by Exploiting Finer-Grained Parallelism
Dryden, N., Maruyama, N., Benson, T., Moon, T., Snir, M., & Essen, B. V. (2019) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2019.00031 · Cited by 44
Parallel Deep Neural Network Training for Big Data on Blue Gene/Q
Chung, I., Sainath, T. N., Ramabhadran, B., Picheny, M., Gunnels, J. A., Austel, V., Chauhari, U., & Kingsbury, B. E. D. (2016) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2016.2626289 · Cited by 43
AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural NetworksPreprint
Gaunt, A. L., Johnson, M., Riechert, M., Tarlow, D., Tomioka, R., Vytiniotis, D., & Webster, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09786 · Cited by 42
Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple MachinesPreprint
Zhang, H., Hu, Z., Wei, J., Xie, P., Kim, G., Ho, Q., & Xing, E. P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.06216 · Cited by 39
100-epoch ImageNet Training with AlexNet in 24 Minutes.Preprint
You, Y., Zhang, Z., Hsieh, C., & Demmel, J. W. (2017) · arXiv (Cornell University) · Cited by 36
Energy, Memory, and Runtime Tradeoffs for Implementing Collective Communication Operations
Hoefler, T., & Moor, D. (2014) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000093896 · Cited by 34
Self-citationEase.ml: Towards Multi-tenant Resource Sharing for Machine Learning WorkloadsPreprint
Li, T., Zhong, J., Liu, J., Wu, W., & Zhang, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.07308 · Cited by 32
DeepSpark: Spark-Based Deep Learning Supporting Asynchronous Updates and Caffe CompatibilityPreprint
Kim, H., Park, J. H., Jang, J., & Yoon, S. (2016) · arXiv (Cornell University) · Cited by 30
Oyama, Y., Nomura, A., Sato, I., Nishimura, H., Tamatsu, Y., & Matsuoka, S. (2016) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2016.7840590 · Cited by 30
Asynchronous parallel stochastic gradient descent
Keuper, J., & Pfreundt, F. (2015) · doi:10.1145/2834892.2834893 · Cited by 28
Distributed learning of multilingual DNN feature extractors using GPUs
Miao, Y., Zhang, H., & Metze, F. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-211 · Cited by 26
CHAOS: a parallelization scheme for training convolutional neural networks on Intel Xeon Phi
Viebke, A., Memeti, S., Sabri Pllana, A., & Abraham, A. (2017) · The Journal of Supercomputing · doi:10.1007/s11227-017-1994-x · Cited by 25
Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific DataPreprint
Kurth, T., Zhang, J., Nadathur, S., Mitliagkas, I., Racah, E., Patwary, M. A., Malas, T. B., Sundaram, N., Bhimji, W., Smorkalov, M. E., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05256 · Cited by 24
Accelerating Deep Learning Frameworks with Micro-Batches
Oyama, Y., Ben‐Nun, T., Hoefler, T., & Matsuoka, S. (2018) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster.2018.00058 · Cited by 24
Self-citationSecond-order Optimization Method for Large Mini-batch: Training ResNet-50 on ImageNet in 35 Epochs
Osawa, K., Tsuji, Y., Ueno, Y., Naruse, A., Yokota, R., & Matsuoka, S. (2018) · Cited by 23
Miikkulainen, R. P. (2017) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3067695.3067716 · Cited by 21
ZNNi: maximizing the inference throughput of 3D convolutional networks on CPUs and GPUs
Zlateski, A., Lee, K., & Seung, H. S. (2016) · IEEE International Conference on High Performance Computing, Data, and Analytics · doi:10.5555/3014904.3015002 · Cited by 20
Cohen, N., Sharir, O., & Shashua, A. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · Cited by 20
Lighter-communication distributed machine learning via Sufficient Factor Broadcasting
Xie, P., Kim, J. K., Zhou, Y. X., Ho, Q., Kumar, A., Yu, Y., & Xing, E. P. (2016) · Uncertainty in Artificial Intelligence · Cited by 17
Large Scale Language Modeling: Converging on 40GB of Text in Four Hours
Puri, R., Kirby, R. M., Yakovenko, N., & Catanzaro, B. (2018) · International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD) · doi:10.1109/cahpc.2018.8645935 · Cited by 13
GaDei: On Scale-Up Training as a Service for Deep Learning
Zhang, W., Feng, M., Zheng, Y., Ren, Y., Wang, Y., Liu, J., Liu, P., Xiang, B., Zhang, L., Zhou, B., et al. (2017) · Proceedings · doi:10.1109/icdm.2017.161 · Cited by 10
Using Supercomputer to Speed up Neural Network Training
Yu, Y., Jiang, J., & Chi, X. (2016) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads.2016.0126 · Cited by 7
Instant Learning: Parallel Deep Neural Networks and Convolutional BootstrappingPreprint
Simpson, A. J. R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.05972 · Cited by 4
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1609.07061
Hoffer, E., Hubara, I., & Soudry, D. (2017) · arXiv · doi:10.48550/arxiv.1705.08741
Cited by
The future of digital health with federated learning
Rieke, N., Hancox, J., Li, W., Milletarì, F., Roth, H. R., Albarqouni, S., Bakas, S., Galtier, M. N., Landman, B. A., Maier-Hein, K., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00323-1 · Cited by 4,015
Parallel convolutional processing using an integrated photonic tensor core
Feldmann, J., Youngblood, N., Karpov, M., Gehring, H., Li, X., Stappers, M., Le Gallo, M., Fu, X., Lukashchuk, A., Raja, A. S., et al. (2021) · Nature · doi:10.1038/s41586-020-03070-1 · Cited by 1,900
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994
Federated Learning: A Survey on Enabling Technologies, Protocols, and Applications
Aledhari, M., Razzak, R., Parizi, R. M., & Saeed, F. (2020) · IEEE Access · doi:10.1109/access.2020.3013541 · Cited by 733
Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., & Papernot, N. (2021) · 2021 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40001.2021.00019 · Cited by 810
“Instead, in typical distributed training algorithms [38], [39], training updates from each constituent are shared between each other and training iterates based on a joint update .”
Background
Metadata
- Authors
- Ben-Nun, T., & Hoefler, T.
- Venue
- ACM Computing Surveys
- Published
- 2019-08-30
- Volume
- 52
- Issue
- 4
- Pages
- 1-43
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2972087877
- Cited by
- 577
- References
- 255
- FWCI
- 81.27
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
