The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
GeePS: scalable deep learning on distributed GPUs with a GPU-specialized parameter server
Open accessAbstract
Large-scale deep learning requires huge computational resources to train a multi-layer neural network. Recent systems propose using 100s to 1000s of machines to train networks with tens of layers and billions of connections. While the computation involved can be done more efficiently on GPUs than on more traditional CPU cores, training such networks on a single GPU is too slow and training on distributed GPUs can be inefficient, due to data movement overheads, GPU stalls, and limited GPU memory. This paper describes a new parameter server, called GeePS, that supports scalable deep learning across GPUs distributed among multiple machines, overcoming these obstacles. We show that GeePS enables a state-of-the-art single-node GPU implementation to scale well, such as to 13 times the number of training images processed per second on 16 machines (relative to the original optimized single-node code). Moreover, GeePS achieves a higher training throughput with just four GPU machines than that a state-of-the-art CPU-only system achieves with 108 machines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575
Multi-column deep neural networks for image classification
Ciresan, D., Meier, U., & Schmidhuber, J. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248110 · Cited by 3,720
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,199
Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition
Dahl, G. E., Yu, D., Deng, L., & Acero, A. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2134090 · Cited by 3,011 · cited 1×
“Large-scale deep learning is emerging as a primary machine learning approach for important, challenging problems such as image classification [7, 14, 23, 33] and speech recognition [13, 18].”
BackgroundBeyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299101 · Cited by 2,062
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., & Zhang, Z. (2015) · arXiv · doi:10.48550/arxiv.1512.01274 · Cited by 1,819
Going Deeper with Convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014) · arXiv · doi:10.48550/arxiv.1409.4842 · Cited by 1,461
Scaling distributed machine learning with the parameter server
Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed, A. A., Josifovski, V., Long, J. E., Shekita, E. J., & Su, B.-Y. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685095 · Cited by 1,021
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. (2014) · doi:10.21236/ada623249 · Cited by 968
(2007) · Cited by 937
One weird trick for parallelizing convolutional neural networks
Krizhevsky, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1404.5997 · Cited by 919
(2013) · Cited by 651
Project Adam: building an efficient and scalable deep learning training system
Chilimbi, T. M., Suzue, Y., Apacible, J., & Kalyanaraman, K. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685094 · Cited by 550
More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server
Ho, Q., Cipar, J., Cui, H., Kim, J. K., Lee, S., Gibbons, P. B., Gibson, G. A., Ganger, G. R., & Xing, E. P. (2013) · PubMed · doi:10.1184/r1/6475898 · Cited by 506
Self-citationAsynchronous Distributed ADMM for Consensus Optimization
Zhang, R., & Kwok, J. T.-Y. (2014) · Cited by 335
Deep Image: Scaling up Image Recognition
Wu, R., Yan, S., Shan, Y., Dang, Q., & Sun, G. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1501.02876 · Cited by 313
Piccolo: building fast, distributed programs with partitioned tables
Power, R., & Li, J. (2010) · doi:10.5555/1924943.1924964 · Cited by 252
Scalable inference in latent variable models
Ahmed, A., Aly, M., Gonzalez, J., Narayanamurthy, S., & Smola, A. J. (2012) · Proceedings of the fifth ACM international conference on Web search and data mining · doi:10.1145/2124295.2124312 · Cited by 235
Exploiting bounded staleness to speed up Big Data analytics
Cui, H., Cipar, J., Ho, Q., Kim, J. K., Lee, S., Kumar, A., Wei, J., Dai, W., Ganger, G. R., Gibbons, P. B., et al. (2018) · doi:10.1184/r1/6475613 · Cited by 119
Self-citationManaged communication and consistency for fast data-parallel iterative analytics
Wei, J., Dai, W., Qiao, A., Ho, Q., Cui, H., Ganger, G. R., Gibbons, P. B., Gibson, G. A., & Xing, E. P. (2015) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/2806777.2806778 · Cited by 104
Self-citationDistributed Stochastic Gradient MCMC
Ahn, S., Shahbaba, B., & Welling, M. (2014) · UvA-DARE (University of Amsterdam) · Cited by 72
Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple MachinesPreprint
Zhang, H., Hu, Z., Wei, J., Xie, P., Kim, G., Ho, Q., & Xing, E. P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.06216 · Cited by 39
Self-citationMinerva: A Scalable and Highly Efficient Training Platform for Deep Learning
Wang, M., Xiao, T., Li, J., Zhang, J., Hong, C., & Zhang, Z. (2014) · neural information processing systems · Cited by 18
Cited by
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Deep Learning in Mobile and Wireless Networking: A Survey
Zhang, C., Patras, P., & Haddadi, H. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2904897 · Cited by 1,768
Deep learning for universal linear embeddings of nonlinear dynamics
Lusch, B., Kutz, J. N., & Brunton, S. L. (2018) · Nature Communications · doi:10.1038/s41467-018-07210-0 · Cited by 1,424
Next generation cloud computing: New trends and research directions
Varghese, B., & Buyya, R. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2017.09.020 · Cited by 857
PipeDream: generalized pipeline parallelism for DNN training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., Gibbons, P. B., & Zaharia, M. (2019) · Proceedings of the 27th ACM Symposium on Operating Systems Principles · doi:10.1145/3341301.3359646 · Cited by 818
Self-citation
Metadata
- Authors
- Cui, H., Zhang, H., Ganger, G. R., Gibbons, P. B., & Xing, E. P.
- Venue
- Proceedings of the Eleventh European Conference on Computer Systems
- Publisher
- ACM
- Published
- 2016-04-18
- Pages
- 1-16
- OpenAlex
- W2339765813
- Cited by
- 286
- References
- 32
- FWCI
- 31.73
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.