The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Petuum: A New Platform for Distributed Machine Learning on Big Data
Open accessAbstract
What is a systematic way to efficiently apply a wide spectrum of advanced ML programs to industrial scale problems, using Big Models (up to 100 s of billions of parameters) on Big Data (up to terabytes or petabytes)? Modern parallelization strategies employ fine-grained operations and scheduling beyond the classic bulk-synchronous processing paradigm popularized by MapReduce, or even specialized graph-based execution that relies on graph representations of ML programs. The variety of approaches tends to pull systems and algorithms design in different directions, and it remains difficult to find a universal platform applicable to a wide range of ML programs at scale. We propose a general-purpose framework, Petuum, that systematically addresses data- and model-parallel challenges in large-scale ML, by observing that many ML programs are fundamentally optimization-centric and admit error-tolerant, iterative-convergent algorithmic solutions. This presents unique opportunities for an integrative system design, such as bounded-error network synchronization and dynamic scheduling based on ML program structure. We demonstrate the efficacy of these system designs versus well-known implementations of modern ML algorithms, showing that Petuum allows ML programs to run in much less time and at considerably larger model sizes, even on modestly-sized compute clusters.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
(2012) · Cited by 9,062
Griffiths, T. L., & Steyvers, M. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0307752101 · Cited by 6,049
(2010) · Cited by 4,411
Pregel: a system for large-scale graph processing
Malewicz, G., Austern, M. H., Bik, A. J. C., Dehnert, J. C., Horn, I., Leiser, N., & Czajkowski, G. (2010) · Proceedings of the 2010 ACM SIGMOD International Conference on Management of data · doi:10.1145/1807167.1807184 · Cited by 3,515
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,199
White, T. (2009) · Cited by 2,800
Distance Metric Learning with Application to Clustering with Side-Information
Xing, E. P., Jordan, M. I., Russell, S., & Ng, A. Y. (2002) · Cited by 2,700
Self-citationInformation-theoretic metric learning
Davis, J. V., Kulis, B., Jain, P., Sra, S., & Dhillon, I. S. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273523 · Cited by 2,006
Building high-level features using large scale unsupervised learning
Le, Q. V. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639343 · Cited by 1,695
Keulen, D. S. (2014) · Cited by 1,677
Distributed GraphLab: a framework for machine learning and data mining in the cloud
Low, Y., Bickson, D., Gonzalez, J., Guestrin, C., Kyrola, A., & Hellerstein, J. M. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2212351.2212354 · Cited by 1,653
Stochastic variational inference
Hoffman, M. D., Blei, D. M., Wang, C., & Paisley, J. W. (2013) · Journal of Machine Learning Research · doi:10.5555/2567709.2502622 · Cited by 1,333
Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
Niu, F., Recht, B., Re, C., & Wright, S. J. (2011) · arXiv · doi:10.48550/arxiv.1106.5730 · Cited by 1,211
Scaling distributed machine learning with the parameter server
Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed, A. A., Josifovski, V., Long, J. E., Shekita, E. J., & Su, B.-Y. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685095 · Cited by 1,021
Scaling Distributed Machine Learning with the Parameter Server
Li, M. (2014) · Proceedings of the 2014 International Conference on Big Data Science and Computing · doi:10.1145/2640087.2644155 · Cited by 979
Ad click prediction: a view from the trenches
McMahan, H. B., Holt, G., Sculley, D., Young, M., Ebner, D., Grady, J., Nie, L., Phillips, T., Davydov, E., Golovin, D., et al. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2488200 · Cited by 867
Large-Scale Parallel Collaborative Filtering for the Netflix Prize
Zhou, Y., Wilkinson, D., Schreiber, R., & Pan, R. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-68880-8_32 · Cited by 700
Building high-level features using large scale unsupervised learning
Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G. S., Dean, J., Le, Q. V., & Ng, A. Y. (2012) · International Conference on Machine Learning · Cited by 646
Large-scale matrix factorization with distributed stochastic gradient descent
Gemulla, R., Nijkamp, E., Haas, P. J., & Sismanis, Y. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020426 · Cited by 589
More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server
Ho, Q., Cipar, J., Cui, H., Kim, J. K., Lee, S., Gibbons, P. B., Gibson, G. A., Ganger, G. R., & Xing, E. P. (2013) · PubMed · doi:10.1184/r1/6475898 · Cited by 506
Self-citationEfficient methods for topic model inference on streaming document collections
Yao, L., Mimno, D., & McCallum, A. (2009) · Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1557019.1557121 · Cited by 410
Parallel coordinate descent methods for big data optimization
Richtárik, P., & Takáč, M. (2016) · Mathematical Programming · doi:10.1007/s10107-015-0901-6 · Cited by 396
Scalable Coordinate Descent Approaches to Parallel Matrix Factorization for Recommender Systems
Yu, H.-F., Hsieh, C.-J., Si, S., & Dhillon, I. (2012) · 2012 IEEE 12th International Conference on Data Mining · doi:10.1109/icdm.2012.168 · Cited by 253
Piccolo: building fast, distributed programs with partitioned tables
Power, R., & Li, J. (2010) · doi:10.5555/1924943.1924964 · Cited by 252
Distributed delayed stochastic optimization
Agarwal, A., & Duchi, J. C. (2012) · 2012 IEEE 51st IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.2012.6426626 · Cited by 243
Scalable inference in latent variable models
Ahmed, A., Aly, M., Gonzalez, J., Narayanamurthy, S., & Smola, A. J. (2012) · Proceedings of the fifth ACM international conference on Web search and data mining · doi:10.1145/2124295.2124312 · Cited by 235
Langford, J. C., Smola, A. J., & Zinkevich, M. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0911.0491 · Cited by 178
LightLDA: Big Topic Models on Modest Computer Clusters
Yuan, J., Gao, F., Ho, Q., Dai, W., Wei, J., Zheng, X., Xing, E. P., Liu, T., & Ma, W. (2015) · International World Wide Web Conference (WWW) · doi:10.1145/2736277.2741115 · Cited by 158
Self-citationPrIter: a distributed framework for prioritized iterative computations
Zhang, Y., Gao, Q., Gao, L., & Wang, C. (2011) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/2038916.2038929 · Cited by 130
High-Performance Distributed ML at Scale through Parameter Server Consistency Models
Dai, W., Kumar, A., Wei, J., Ho, Q., Gibson, G. A., & Xing, E. P. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9195 · Cited by 109
Self-citationParallel Coordinate Descent for L1-Regularized Loss Minimization
Kyrola, A., Bickson, D., Guestrin, C., & Bradley, J. K. (2011) · International Conference on Machine Learning · Cited by 99
Upper and Lower Bounds on the Learning Curve for Gaussian Processes
Williams, C. K. I., & Vivarelli, F. (2000) · Machine Learning · doi:10.1023/a:1007601601278 · Cited by 57
Fugue: Slow-Worker-Agnostic Distributed Learning for Big Models on Big Data
Kumar, A., Beutel, A., Ho, Q., & Xing, E. P. (2018) · Figshare · doi:10.1184/r1/6475643 · Cited by 12
Self-citation
Cited by
A survey on federated learning
Zhang, C., Xie, Y., Bai, H., Yu, B., Li, W., & Gao, Y. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106775 · Cited by 1,899
Deep Learning in Mobile and Wireless Networking: A Survey
Zhang, C., Patras, P., & Haddadi, H. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2904897 · Cited by 1,768
Exploiting Unintended Feature Leakage in Collaborative Learning
Melis, L., Song, C., De Cristofaro, E., & Shmatikov, V. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00029 · Cited by 1,734
Explainable AI (XAI): Core Ideas, Techniques, and Solutions
Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., et al. (2023) · ACM Computing Surveys · doi:10.1145/3561048 · Cited by 1,386
Machine learning on big data: Opportunities and challenges
Zhou, L., Pan, S., Wang, J., & Vasilakos, A. V. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.01.026 · Cited by 1,228
Metadata
- Authors
- Xing, E. P., Ho, Q., Dai, W., Kim, J. K., Wei, J., Lee, S., Zheng, X., Xie, P., Kumar, A., & Yu, Y.
- Venue
- IEEE Transactions on Big Data
- Published
- 2015-06-01
- Volume
- 1
- Issue
- 2
- Pages
- 49-67
- ISSN
- 2332-7790 (ISSN-L)
- OpenAlex
- W2579247884
- Cited by
- 388
- References
- 34
- FWCI
- 105.41
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.