The DOI matches its OpenAlex record; 39 of the 39 references it cites were found in a registry or an index.
Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
Open accessOverview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
Distributed Subgradient Methods for Multi-Agent Optimization
Nedic, A., & Ozdaglar, A. (2009) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2008.2009515 · Cited by 3,719
Robust Stochastic Approximation Approach to Stochastic Programming
Nemirovski, A., Juditsky, A., Lan, G., & Shapiro, A. (2009) · SIAM Journal on Optimization · doi:10.1137/070704277 · Cited by 1,945
A Structured Self-attentive Sentence Embedding
Lin, Z., Feng, M., Santos, C. N. D., Yu, M., Xiang, B., Zhou, B., & Bengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.03130 · Cited by 1,743 · cited 1×
“For image classification we train ResNet [He et al., 2015] with different number of layers on CIFAR-10 [Krizhevsky, 2009]; for speech recognition, we train both proprietary and public dataset on a proprietary CNN model that we get from our industry partner [Feng et al., 2016, Lin et al., 2017].”
MethodOnline Passive-Aggressive Algorithms
Crammer, K., Dekel, O., Keshet, J., Shalev‐Shwartz, S., & Singer, Y. (2003) · Cited by 1,665
Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
Niu, F., Recht, B., Re, C., & Wright, S. J. (2011) · arXiv · doi:10.48550/arxiv.1106.5730 · Cited by 1,211 · cited 2×
“ntain a linear speedup on convex, strongly convex and nonconvex objectives when the staleness of the gradient is bounded [Agarwal and Duchi, 2011, Feyzmahdavian et al., 2015, Lian et al., 2015, 2016, Recht et al., 2011]. Decentralized parallel stochastic algorithms There are existing work on decentralized parallel sto…”
BackgroundMethodParallelized Stochastic Gradient Descent
Zinkevich, M., Weimer, M., Li, L., & Smola, A. J. (2010) · Cited by 1,124
Online Learning and Online Convex Optimization
Shalev-Shwartz, S. (2011) · now publishers, Inc. eBooks · doi:10.1561/9781601985477 · Cited by 1,095
Scaling distributed machine learning with the parameter server
Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed, A. A., Josifovski, V., Long, J. E., Shekita, E. J., & Su, B.-Y. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685095 · Cited by 1,021
Distributed Stochastic Subgradient Projection Algorithms for Convex Optimization
Sundhar Ram, S., Nedić, A., & Veeravalli, V. V. (2010) · Journal of Optimization Theory and Applications · doi:10.1007/s10957-010-9737-7 · Cited by 814 · cited 6×
“…and Nedic, 2011, Sundhar Ram et al., 2010] or implicitly indicate that decentralized algorithms were much worse than centralized algorithms in terms of computational complexity and total communication complexity [Aybat et al., 2015, Lan et al., 2017, Ram et al., 2010, Zhang and Kwok, 2014].”
BackgroundMethodInfluentialOn the Convergence of Decentralized Gradient Descent
Yuan, K., Ling, Q., & Yin, W. (2013) · arXiv · doi:10.48550/arxiv.1310.7063 · Cited by 635
On the Convergence of Decentralized Gradient Descent
Yuan, K., Ling, Q., & Yin, W. (2016) · SIAM Journal on Optimization · doi:10.1137/130943170 · Cited by 633
Gossip algorithms: design, analysis and applications
Boyd, S., Ghosh, A., Prabbakar, B., & Shah, D. (2005) · Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies. · doi:10.1109/infcom.2005.1498447 · Cited by 558
Revisiting Distributed Synchronous SGD
Pan, X., Chen, J., Monga, R., Bengio, S., & Józefowicz, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.05800 · Cited by 548
Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning
Bach, F., & Moulines, É. (2011) · Cited by 488
Broadcast Gossip Algorithms for Consensus
Aysal, T. C., Yildiz, M. E., Sarwate, A. D., & Scaglione, A. (2009) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2009.2016247 · Cited by 480
Asynchronous Distributed ADMM for Consensus Optimization
Zhang, R., & Kwok, J. T.-Y. (2014) · International Conference on Machine Learning · Cited by 439
Distributed Delayed Stochastic OptimizationPreprint
Agarwal, A., & Duchi, J. C. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1104.5525 · Cited by 385
Randomized consensus algorithms over large scale networks
Fagnani, F., & Zampieri, S. (2008) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2008.080506 · Cited by 350
Distributed Asynchronous Constrained Stochastic Optimization
Srivastava, K., & Nedic, A. (2011) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2011.2118740 · Cited by 350
Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
Lian, X., Huang, Y., Li, Y., & Liu, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.08272 · Cited by 324
Self-citationDistributed Autonomous Online Learning: Regrets and Intrinsic Privacy-Preserving Properties
Yan, F., Sundaram, S., Vishwanathan, S. V. N., & Qi, Y. (2013) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2012.191 · Cited by 263
Gossip consensus algorithms via quantized communication
Carli, R., Fagnani, F., Frasca, P., & Zampieri, S. (2010) · Automatica · doi:10.1016/j.automatica.2009.10.032 · Cited by 239
A distributed consensus protocol for clock synchronization in wireless sensor network
Schenato, L., & Gamba, G. (2007) · 2007 46th IEEE Conference on Decision and Control · doi:10.1109/cdc.2007.4434671 · Cited by 224
Communication-efficient algorithms for decentralized and stochastic optimization
Lan, G., Lee, S., & Zhou, Y. (2020) · Mathematical Programming · doi:10.1007/s10107-018-1355-4 · Cited by 187
A fast parallel SGD for matrix factorization in shared memory systems
Zhuang, Y., Chin, W.-S., Juan, Y.-C., & Lin, C. (2013) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2507157.2507164 · Cited by 165
DSA: Decentralized Double Stochastic Averaging Gradient AlgorithmPreprint
Mokhtari, A., & Ribeiro, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.04216 · Cited by 148
An Asynchronous Mini-Batch Algorithm for Regularized Stochastic Optimization
Feyzmahdavian, H. R., Aytekin, A., & Johansson, M. (2016) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2016.2525015 · Cited by 136
Performance of a Distributed Stochastic Approximation Algorithm
Bianchi, P., Fort, G., & Hachem, W. (2013) · IEEE Transactions on Information Theory · doi:10.1109/tit.2013.2275131 · Cited by 122
Asynchronous gossip algorithms for stochastic optimization
Ram, S. S., Nedić, A., & Veeravalli, V. V. (2009) · International Conference on Game Theory for Networks · doi:10.1109/gamenets.2009.5137386 · Cited by 94
Lian, X., Zhang, H., Hsieh, C., Huang, Y., & Liu, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00498 · Cited by 91
Self-citationAsynchronous Gossip Algorithm for Stochastic Optimization: Constant Stepsize Analysis*
Ram, S. S., Nedić, A., & Veeravalli, V. V. (2010) · Springer eBooks · doi:10.1007/978-3-642-12598-0_5 · Cited by 48
Distributed subgradient projection algorithm for convex optimization
Ram, S. S., Nedić, A., & Veeravalli, V. V. (2009) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2009.4960418 · Cited by 48
A gossip algorithm for convex consensus optimization over networks
Lu, J., Tang, C. Y., Regier, P. R., & Bow, T. D. (2010) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.1109/acc.2010.5530844 · Cited by 21
Distributed Deep Learning for Question Answering
Feng, M., Xiang, B., & Zhou, B. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983377 · Cited by 10
Decentralized Consensus Optimization with Asynchrony and DelaysPreprint
Wu, T., Yuan, K., Ling, Q. H., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.00150 · Cited by 9
Cited by
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772
Federated learning: Challenges, methods, and future directions
Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2975749 · Cited by 6,210
“For instance, numerous methods have been proposed to tackle expensive communication in the optimization and signal processing communities [28], [40], [43].”
MethodFederated Learning With Differential Privacy: Algorithms and Performance Analysis
Wei, K., Li, J., Ding, M., Ma, C., Yang, H. H., Farokhi, F., Jin, S., Quek, T. Q. S., & Vincent Poor, H. (2020) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2020.2988575 · Cited by 2,400
“The work in [18] considered partially global aggregation, where after each local update step, parameter”
BackgroundAdaptive Federated Learning in Resource Constrained Edge Computing Systems
Wang, S., Tuor, T., Salonidis, T., Leung, K. K., Makaya, C., He, T., & Chan, K. (2019) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2019.2904348 · Cited by 2,232
A survey on security and privacy of federated learning
Mothukuri, V., Parizi, R. M., Pouriyeh, S., Huang, Y., Dehghantanha, A., & Srivastava, G. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2020.10.007 · Cited by 1,480
Metadata
- Authors
- Lian, X., Zhang, C., Zhang, H., Hsieh, C., Zhang, W., & Liu, J.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2017-05-25
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1705.09056
- OpenAlex
- W2617960902
- Cited by
- 395
- References
- 39
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.