Revisiting Distributed Synchronous SGD
Open accessThe DOI matches its OpenAlex record; 26 of the 26 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
(2010) · Cited by 8,261
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
Self-citationDeep Neural Networks for Acoustic Modeling in Speech Recognition
Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. (2012) · IEEE Signal Processing Magazine · Cited by 2,136
Dean, J., & Barroso, L. A. (2013) · Communications of the ACM · doi:10.1145/2408776.2408794 · Cited by 1,852 · cited 1×
“To alleviate the straggler problem, we introduce backup workers (Dean & Barroso, 2013) as follows: instead of having only N workers, we add b extra workers, but as soon as the parameter servers receive gradients from any N workers, they stop waiting and update their parameters using the N gradients.”
MethodConditional Image Generation with PixelCNN Decoders
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05328 · Cited by 1,833
Cited by
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Self-citationLearning Transferable Architectures for Scalable Image Recognition
Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00907 · Cited by 7,831
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215
“[8] show this approach is state-of-the-art in the data center setting, where it outperforms asynchronous approaches.”
BackgroundPractical Secure Aggregation for Privacy-Preserving Machine Learning
Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., Ramage, D., Segal, A., & Seth, K. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3133982 · Cited by 3,956
“…consists of finding parameters Θ that achieve small L f ( D , Θ ) , typically by iterating a variant of a minibatch stochastic gradient descent rule [21], [22]: where Θ t are the parameters after iteration t , D t ⊆ D is a randomly selected subset of the training examples, and η is a learning rate…”
BackgroundFederated Learning in Mobile Edge Networks: A Comprehensive Survey
Lim, W. Y. B., Luong, N. C., Hoang, D. T., Jiao, Y., Liang, Y.-C., Yang, Q., Niyato, D., & Miao, C. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2986024 · Cited by 2,565
“To account for small gradient values that can delay convergence if they are ignored and not updated completely [108], these gradient values are accumulated as residual values.”
BackgroundAccurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294
Adaptive Federated Learning in Resource Constrained Edge Computing Systems
Wang, S., Tuor, T., Salonidis, T., Leung, K. K., Makaya, C., He, T., & Chan, K. (2019) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2019.2904348 · Cited by 2,232
Federated Learning with Non-IID Data
Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., & Chandra, V. (2018) · arXiv · doi:10.48550/arxiv.1806.00582 · Cited by 2,077
Deep Learning With Edge Computing: A Review
Chen, J., & Ran, X. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2921977 · Cited by 1,540
Self-citationHow to backdoor federated learning
Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., & Shmatikov, V. (2018) · arXiv · doi:10.48550/arxiv.1807.00459 · Cited by 1,495
“In comparison to synchronous distributed SGD [6], federated learning reduces the number of participants per round and converges faster.”
BackgroundInfluentialFederated Learning for Internet of Things: Recent Advances, Taxonomy, and Open Challenges
Khan, L. U., Saad, W., Han, Z., Hossain, E., & Hong, C. S. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3090430 · Cited by 892
Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning
Wang, Z., Song, M., Zhang, Z., Song, Y., Wang, Q., & Qi, H. (2019) · IEEE INFOCOM 2019 - IEEE Conference on Computer Communications · doi:10.1109/infocom.2019.8737416 · Cited by 881
A Survey on Distributed Machine Learning
Verbraeken, J., Wolting, M., Katzy, J., Kloppenburg, J., Verbelen, T., & Rellermeyer, J. S. (2021) · ACM Computing Surveys · doi:10.1145/3377454 · Cited by 872
PipeDream: generalized pipeline parallelism for DNN training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., Gibbons, P. B., & Zaharia, M. (2019) · Proceedings of the 27th ACM Symposium on Operating Systems Principles · doi:10.1145/3341301.3359646 · Cited by 818
Multi-task Self-Supervised Visual Learning
Doersch, C., & Zisserman, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.226 · Cited by 598
Distributed Learning in Wireless Networks: Recent Progress and Future Challenges
Chen, M., Gunduz, D., Huang, K., Saad, W., Bennis, M., Feljan, A. V., & Poor, H. V. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3118346 · Cited by 548
Split learning for health: Distributed deep learning without sharing raw patient data
Vepakomma, P., Gupta, O., Swedish, T., & Raskar, R. (2018) · arXiv · doi:10.48550/arxiv.1812.00564 · Cited by 513
“In addition to splitNN [32], techniques of federated deep learning [1] and large batch synchronous stochastic gradient descent (SGD)[19] are currently available approaches for distributed deep learning.”
MethodLarge Batch Training of Convolutional Networks
You, Y., Gitman, I., & Ginsburg, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.03888 · Cited by 463
Local Differential Privacy-Based Federated Learning for Internet of Things
Zhao, Y., Zhao, J., Yang, M., Wang, T., Wang, N., Lyu, L., Niyato, D., & Lam, K.-Y. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3037194 · Cited by 456
“• We propose a local differential privacy based federated learning training vehicular crowdsourcing system, and an LDP-FedSGD algorithm by taking advantage of FL and LDP. • Experimental evaluation of our proposed mechanisms on real-world datasets and synthetic datasets demonstrates that our proposed mechanisms achieve…”
MethodInfluentialThe Best of Both Worlds: Combining Recent Advances in Neural Machine Translation
Chen, M. X., Firat, O., Bapna, A., Johnson, M., Macherey, W., Foster, G., Jones, L., Schuster, M., Shazeer, N., Parmar, N., et al. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1008 · Cited by 564
A Survey of Recent Advances in Edge-Computing-Powered Artificial Intelligence of Things
Chang, Z., Liu, S., Xiong, X., Cai, Z., & Tu, G. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3088875 · Cited by 418
Clipper: A Low-Latency Online Prediction Serving System
Crankshaw, D., Wang, X., Zhou, G., Franklin, M. J., Gonzalez, J. E., & Stoica, I. (2016) · arXiv · doi:10.48550/arxiv.1612.03079 · Cited by 415
Optimus: an efficient dynamic resource scheduler for deep learning clusters
Peng, Y., Bao, Y., Chen, Y., Wu, C., & Guo, C. (2018) · Proceedings of the Thirteenth EuroSys Conference · doi:10.1145/3190508.3190517 · Cited by 409
Distributed Artificial Intelligence Empowered by End-Edge-Cloud Computing: A Survey
Duan, S., Wang, D., Ren, J., Lyu, F., Zhang, Y., Wu, H., & Shen, X. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3218527 · Cited by 408
Privacy preservation in federated learning: An insightful survey from the GDPR perspective
Truong, N., Sun, K., Wang, S., Guitton, F., & Guo, Y. (2021) · Computers & Security · doi:10.1016/j.cose.2021.102402 · Cited by 399
Lian, X., Zhang, C., Zhang, H., Hsieh, C., Zhang, W., & Liu, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09056 · Cited by 395
Thapa, C., & Camtepe, S. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104130 · Cited by 393
You, Y., Zhang, Z., Hsieh, C.-J., Demmel, J., & Keutzer, K. (2018) · Proceedings of the 47th International Conference on Parallel Processing · doi:10.1145/3225058.3225069 · Cited by 384
Zhu, G., Du, Y., Gunduz, D., & Huang, K. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2020.3039309 · Cited by 376
“The first is to discard the updates from slow-responding edge devices (stragglers) for fast update synchronization [8], [9].”
BackgroundBatch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
Ioffe, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.03275 · Cited by 367
Itahara, S., Nishio, T., Koda, Y., Morikura, M., & Yamamoto, K. (2020) · arXiv · doi:10.48550/arxiv.2008.06180 · Cited by 338
Itahara, S., Nishio, T., Koda, Y., Morikura, M., & Yamamoto, K. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2021.3070013 · Cited by 338
“In this approach, network traffic reduces drastically compared to that of algorithms considering iterating local model update andmodel parameter exchange alternately[13].”
MethodThe Reversible Residual Network: Backpropagation Without Storing Activations
Gomez, A. N., Ren, M., Urtasun, R., & Grosse, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.04585 · Cited by 335
Revisiting Small Batch Training for Deep Neural Networks
Masters, D. A., & Luschi, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.07612 · Cited by 333
Multi-Armed Bandit-Based Client Scheduling for Federated Learning
Xia, W., Quek, T. Q. S., Guo, K., Wen, W., Yang, H. H., & Zhu, H. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2020.3008091 · Cited by 313
Edge Intelligence: Empowering Intelligence to the Edge of Network
Xu, D., Li, T., Li, Y., Su, X., Tarkoma, S., Jiang, T., Crowcroft, J., & Hui, P. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3119950 · Cited by 272
A domain-specific supercomputer for training deep neural networks
Jouppi, N. P., Yoon, D. H., Kurian, G., Li, S., Patil, N., Laudon, J., Young, C., & Patterson, D. (2020) · Communications of the ACM · doi:10.1145/3360307 · Cited by 258
Analyzing User-Level Privacy Attack Against Federated Learning
Song, M., Wang, Z., Zhang, Z., Song, Y., Wang, Q., Ren, J., & Qi, H. (2020) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.3000372 · Cited by 257
Measuring the Effects of Data Parallelism on Neural Network Training
Shallue, C. J., Lee, J., Antognini, J., Sohl-Dickstein, J., Frostig, R., & Dahl, G. E. (2018) · arXiv · doi:10.48550/arxiv.1811.03600 · Cited by 229
When Edge Meets Learning: Adaptive Control for Resource-Constrained Distributed Machine Learning
Wang, S., Tuor, T., Salonidis, T., Leung, K. K., Makaya, C., He, T., & Chan, K. (2018) · IEEE INFOCOM 2018 - IEEE Conference on Computer Communications · doi:10.1109/infocom.2018.8486403 · Cited by 209
Spectral Norm Regularization for Improving the Generalizability of Deep Learning
Yoshida, Y., & Miyato, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10941 · Cited by 205
“we are training a deep neural network using SGD and want to parallelize the process using multiple GPUs or nodes to accelerate the training. A well-known method for achieving this is synchronous SGD [7,3], which requires a large minibatch to effectively exploit parallel computation on multiple GPUs or nodes. However,…”
MethodGmail Smart Compose: Real-Time Assisted Writing
Chen, M. X., Lee, B. N., Bansal, G., Cao, Y., Zhang, S., Lu, J., Tsay, J., Wang, Y., Dai, A. M., Chen, Z., et al. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330723 · Cited by 169
“We use synchronous training [9] for all models.”
MethodCosmoFlow: Using Deep Learning to Learn the Universe at Scale
Mathuriya, A., Bard, D., Mendygral, P., Meadows, L., Arnemann, J., Shao, L., He, S., Karna, T., Moise, D., Pennycook, S. J., et al. (2018) · SC18: International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc.2018.00068 · Cited by 132
Semisupervised Distributed Learning With Non-IID Data for AIoT Service Platform
Chiu, T.-C., Shih, Y.-Y., Pang, A.-C., Wang, C.-S., Weng, W., & Chou, C.-T. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2995162 · Cited by 98
Trusted AI in Multiagent Systems: An Overview of Privacy and Security for Distributed Learning
Ma, C., Li, J., Wei, K., Liu, B., Ding, M., Yuan, L., Han, Z., & Vincent Poor, H. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3306773 · Cited by 96
Wireless Network Intelligence at the Edge
Park, J., Samarakoon, S., Bennis, M., & Debbah, M. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2941458 · Cited by 92
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
Wang, J., Tantia, V., Ballas, N., & Rabbat, M. (2019) · arXiv · doi:10.48550/arxiv.1910.00643 · Cited by 90
“1 INTRODUCTION Distributed optimization (Chen et al., 2016; Goyal et al., 2017) is essential for training large models on large datasets (Radford et al.”
BackgroundA survey for federated learning evaluations: Goals and measures
Chai, D., Wang, L., Yang, L., Zhang, J., Chen, K., & Yang, Q. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3382002 · Cited by 64
“Unlike conventional distributed machine learning, which is carried out on different machines in one data center [35], FL is performed on cross-data-center machines or edge devices, which have lower networking or computing resources [34].”
BackgroundSlow and Stale Gradients Can Win the Race
Dutta, S., Wang, J., & Joshi, G. (2021) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2021.3103770 · Cited by 40
How to Democratise and Protect AI: Fair and Differentially Private Decentralised Deep Learning
Lyu, L., Li, Y., Nandakumar, K., Yu, J., & Ma, X. (2020) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2020.3006287 · Cited by 31
Model poisoning attacks against distributed machine learning systems
Tomsett, R., Chan, K. S., & Chakraborty, S. (2019) · Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications · doi:10.1117/12.2520275 · Cited by 27
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications
Park, J., Samarakoon, S., Elgabli, A., Kim, J., Bennis, M., Kim, S.-L., & Debbah, M. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3055679 · Cited by 25
“To validate its communication efficiency, we compare tripartite SL with the large-scale minibatch stochastic GD (LS-SGD) [190] by measuring their transmitted data until convergence under VGG and ResNet NN model architectures with a medical X-ray data set, CheXpert [191].”
MethodInfluentialTraining Large Neural Networks with Constant Memory using a New Execution Algorithm
Pudipeddi, B., Mesmakhosroshahi, M., Xi, J., & Bharadwaj, S. (2020) · arXiv · doi:10.48550/arxiv.2002.05645 · Cited by 25
A comprehensive survey and taxonomy on privacy-preserving deep learning
Tran, A.-T., Luong, T.-D., & Huynh, V.-N. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127345 · Cited by 21
A Privacy-Preserving Crowd Flow Prediction Framework Based on Federated Learning during Epidemics
Wang, W., Yang, G., Bao, L., Ma, K., & Zhou, H. (2022) · Security and Communication Networks · doi:10.1155/2022/8712597 · Cited by 5
Abdellatif, A. A., Mhaisen, N., Mohamed, A., Erbad, A., Guizani, M., Dawy, Z., & Nasreddine, W. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2021.10.016 · Cited by 4
Gmail Smart Compose: Real-Time Assisted Writing
Chen, M. X., Lee, B. N., Bansal, G., Cao, Y., Zhang, S., Lu, J., Tsay, J., Wang, Y., Dai, A. M., Chen, Z., et al. (2019) · arXiv · doi:10.48550/arxiv.1906.00080
Metadata
- Authors
- Pan, X., Chen, J., Monga, R., Bengio, S., & Józefowicz, R.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2017-02-19
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1702.05800
- OpenAlex
- W2336650964
- Cited by
- 548
- References
- 26
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.