Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training
Open accessThe DOI matches its OpenAlex record; 10 of the 10 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2014) · arXiv · doi:10.48550/arxiv.1412.7062 · Cited by 4,213 · cited 1×
“, when training for semantic image segmentation [5].”
BackgroundAccurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294 · cited 2×
“Assume that: 1) the singe-node training takes Tsingle iterations to complete with an execution time of lsingle taken by each iteration; 2) given a Pipe-SGD cluster with p workers we use the same batch size on each worker as the single-node [12]; 3) the single node and Pipe-SGD train the same epochs on the dataset.”
BackgroundMethodOptimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks
Zhang, C., Li, P., Sun, G., Guan, Y., Xiao, B., & Cong, J. (2015) · Proceedings of the 2015 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/2684746.2689060 · Cited by 2,158 · cited 1×
“The fraction of time spent on communication is further increased when applying accelerators [16, 7, 38, 52, 48, 49, 44], as they decrease computation time while leaving communication time untouched.”
BackgroundDeep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
Lin, Y., Han, S., Mao, H., Wang, Y., & Dally, W. J. (2017) · arXiv · doi:10.48550/arxiv.1712.01887 · Cited by 1,134 · cited 5×
“To obtain the best accuracies for the two large non-convex models such as AlexNet and ResNet, we employ a similiar warm-up scheme as in [33], i.”
BackgroundMethodInfluentialPipeDream: Fast and Efficient Pipeline Parallel DNN Training
Harlap, A., Narayanan, D., Phanishayee, A., Seshadri, V., Devanur, N., Ganger, G., & Gibbons, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03377 · Cited by 153 · cited 3×
“Furthermore, [13] may suffer from staleness of the weight update, which is linear in the number of stages.”
BackgroundMethodInfluential
Cited by
Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning
So, J., Guler, B., & Avestimehr, A. S. (2021) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2021.3054610 · Cited by 318
“Circular communication and training architectures have been considered previously in the context of distributed stochastic gradient descent on clear (unmasked) gradient updates, to reduce communication load [37] or to model dataheterogeneity [38].”
BackgroundData Movement Is All You Need: A Case Study on Optimizing Transformers
Ivanov, A., Dryden, N., Ben-Nun, T., Li, S., & Hoefler, T. (2020) · arXiv · doi:10.48550/arxiv.2007.00072 · Cited by 78
Self-citationCentral Server Free Federated Learning over Single-sided Trust Social Networks
He, C., Tan, C., Tang, H., Qiu, S., & Liu, J. (2019) · arXiv · doi:10.48550/arxiv.1910.04956 · Cited by 60
“Recent rigorous theoretical analysis (Stich, 2018; Wang and Joshi, 2018; Yu et al., 2018; Lin et al., 2018) shows that FedAvg is a special case of averaging periodic SGD (also called local SGD) which allows nodes to perform local updates and infrequent synchronization between them to communicate…”
BackgroundMercury: Efficient On-Device Distributed DNN Training via Stochastic Importance Sampling
Zeng, X., Yan, M., & Zhang, M. (2021) · Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/3485730.3485930 · Cited by 43
“Such reduction can be achieved by either identifying the optimal gradient transfer order [5, 12, 19, 44] or using staled model weights [26].”
MethodDigest: Fast and communication efficient decentralized learning with local updates
Gholami, P., & Seferoglu, H. (2024) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2024.3354236 · Cited by 14
“Asynchronous Gossip algorithms, where nodes communicate asynchronously and without waiting for others are promising to reduce idle nodes and eliminate the stragglers, i.e., delayed nodes [17], [18], [19], [20].”
Background
Metadata
- Authors
- Li, Y., Yu, M., Li, S., Avestimehr, S., Kim, N. S., & Schwing, A. G.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2018-11-08
- Volume
- 31
- Pages
- 8056-8067
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1811.03619
- OpenAlex
- W2962700998
- Cited by
- 60
- References
- 10
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training. https://app.cochat.ai/papers/10.48550/arxiv.1811.03619
- The paper
- Li, Y., Yu, M., Li, S., Avestimehr, S., Kim, N. S., & Schwing, A. G. (2018). Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training. arXiv (Cornell University). https://doi.org/10.48550/arxiv.1811.03619
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.