Modeling and synthesizing task placement constraints in Google compute clusters
The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Evaluating the performance of large compute clusters requires benchmarks with representative workloads. At Google, performance benchmarks are used to obtain performance metrics such as task scheduling delays and machine resource utilizations to assess changes in application codes, machine configurations, and scheduling algorithms. Existing approaches to workload characterization for high performance computing and grids focus on task resource requirements for CPU, memory, disk, I/O, network, etc. Such resource requirements address how much resource is consumed by a task. However, in addition to resource requirements, Google workloads commonly include task placement constraints that determine which machine resources are consumed by tasks. Task placement constraints arise because of task dependencies such as those related to hardware architecture and kernel version.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523
Benchmarking cloud serving systems with YCSB
Cooper, B. F., Silberstein, A., Tam, E., Ramakrishnan, R., & Sears, R. (2010) · Proceedings of the 1st ACM symposium on Cloud computing · doi:10.1145/1807128.1807152 · Cited by 3,716 · cited 1×
“0 benchmarks [30], and the Yahoo! Cloud Servicing Benchmark [11].”
BackgroundGlobus: a Metacomputing Infrastructure Toolkit
Foster, I., & Kesselman, C. (1997) · The International Journal of Supercomputer Applications and High Performance Computing · doi:10.1177/109434209701100205 · Cited by 3,133
Distributed computing in practice: the Condor experience
Thain, D., Tannenbaum, T., & Livny, M. (2005) · Concurrency and Computation: Practice and Experience · doi:10.1002/cpe.938 · Cited by 1,886 · cited 2×
“Examples include: the Condor system [10] that uses the ClassAds mechanism [31], IBM’s load balancer [18], the Utopia system [32], and grid toolkits [15].”
BackgroundMethodGenerating representative Web workloads for network and server performance evaluation
Barford, P., & Crovella, M. (1998) · Proceedings of the 1998 ACM SIGMETRICS joint international conference on Measurement and modeling of computer systems · doi:10.1145/277851.277897 · Cited by 1,431
Web server workload characterization: the search for invariants
Arlitt, M. F., & Williamson, C. L. (1996) · ACM SIGMETRICS Performance Evaluation Review · doi:10.1145/233008.233034 · Cited by 747
Matchmaking: distributed resource management for high throughput computing
Raman, R., Livny, M., & Solomon, M. (2002) · Proceedings. The Seventh International Symposium on High Performance Distributed Computing (Cat. No.98TB100244) · doi:10.1109/hpdc.1998.709966 · Cited by 747 · cited 1×
“How ever, in addition to resource requirements, Google tasks fr equently havetask placement constraints (hereafter, just constraints) similar to the Condor ClassAds mechanism [23].”
BackgroundDRAM errors in the wild: a large-scale field study
Schroeder, B., Pinheiro, E., & Weber, W.-D. (2009) · Proceedings of the eleventh international joint conference on Measurement and modeling of computer systems · doi:10.1145/1555349.1555372 · Cited by 557 · cited 1×
“Examples include: web server workload characterization [1,2,14], scientific and high performance computing workloads [5, 9, 23], chip-level characteristics [22,27], resource consumption of Google compute clus-”
BackgroundThe workload on parallel supercomputers: modeling the characteristics of rigid jobs
Lublin, U., & Feitelson, D. G. (2003) · Journal of Parallel and Distributed Computing · doi:10.1016/s0743-7315(03)00108-4 · Cited by 441
Towards characterizing cloud backend workloads: insights from Google compute clusters
Mishra, A. K., Hellerstein, J. L., Cirne, W., & Das, C. R. (2010) · ACM SIGMETRICS Performance Evaluation Review · doi:10.1145/1773394.1773400 · Cited by 374
Self-citationUtopia: A load sharing facility for large, heterogeneous distributed computer systems
Zhou, S., Zheng, X., Wang, J., & Delisle, P. (1993) · Software Practice and Experience · doi:10.1002/spe.4380231203 · Cited by 345 · cited 1×
“Examples include: the Condor system [10] that uses the ClassAds mechanism [31], IBM’s load balancer [18], the Utopia system [32], and grid toolkits [15].”
MethodAn Analysis of Traces from a Production MapReduce Cluster
Kavulya, S., Tan, J., Gandhi, R., & Narasimhan, P. (2010) · 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing · doi:10.1109/ccgrid.2010.112 · Cited by 315 · cited 3×
“Further, there has been work in the area of workload generation and benchmarking including: MapReduce workload generation [8, 16, 20], Web 1.0 benchmark tools such as ab, httpperf, Web 2.0 benchmarks [30], and the Yahoo!”
InfluentialStatistics-driven workload modeling for the Cloud
Ganapathi, A., Chen, Y., Fox, A., Katz, R., & Patterson, D. (2010) · 2010 IEEE 26th International Conference on Data Engineering Workshops (ICDEW 2010) · doi:10.1109/icdew.2010.5452742 · Cited by 241 · cited 2×
“Further, there has been work in the area of workload generation and benchmarking including: MapReduce workload generation [8, 16, 20], Web 1.”
BackgroundBenchmarks and Standards for the Evaluation of Parallel Job Schedulers
Chapin, S. J., Cirne, W., Feitelson, D. G., Jones, J. P., Leutenegger, S. T., Uwe Schwiegelshohn, A., Smith, W. W., & Talby, D. (1999) · Lecture notes in computer science · doi:10.1007/3-540-47954-6_4 · Cited by 191 · cited 1×
“Examples include: web server workload characterization [1,2,14], scientific and high performance computing workloads [5, 9, 23], chip-level characteristics [22,27], resource consumption of Google compute clus-”
BackgroundAnalysis and Lessons from a Publicly Available Google Cluster Trace
Chen, Y., Ganapathi, A., Griffith, R., & Katz, R. H. (2010) · UC Berkeley · Cited by 151
Characterizing Network Traffic in a Cluster-based, Multi-tier Data Center
Ersoz, D., Yousif, M. S., & Das, C. R. (2007) · Proceedings of the ... International Conference on Distributed Computing Systems/Proceedings of the International Conference on Distributed Computing Systems · doi:10.1109/icdcs.2007.90 · Cited by 127
Self-citationTowards Understanding Cloud Performance Tradeoffs Using Statistical Workload Analysis and Replay
Chen, Y., Ganapathi, A., Griffith, R., & Katz, R. H. (2010) · UC Berkeley · Cited by 46
Cited by
Large-scale cluster management at Google with Borg
Verma, A., Pedrosa, L., Korupolu, M., Oppenheimer, D., Tune, E., & Wilkes, J. (2015) · Proceedings of the Tenth European Conference on Computer Systems · doi:10.1145/2741948.2741964 · Cited by 1,468
Heterogeneity and dynamicity of clouds at scale: Google trace analysis
Reiss, C., Tumanov, A., Ganger, G. R., Katz, R. H., & Kozuch, M. A. (2012) · Proceedings of the Third ACM Symposium on Cloud Computing · doi:10.1145/2391229.2391236 · Cited by 1,063
Omega: flexible, scalable schedulers for large compute clusters
Schwarzkopf, M., Konwinski, A., Abd-El-Malek, M., & Wilkes, J. (2013) · Proceedings of the 8th ACM European Conference on Computer Systems · doi:10.1145/2465351.2465386 · Cited by 650
Sparrow: distributed, low latency scheduling
Ousterhout, K., Wendell, P., Zaharia, M., & Stoica, I. (2013) · Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles · doi:10.1145/2517349.2522716 · Cited by 586
Dynamic energy-aware capacity provisioning for cloud computing environments
Zhang, Q., Zhani, M. F., Zhang, S., Zhu, Q., Boutaba, R., & Hellerstein, J. L. (2012) · Proceedings of the 9th international conference on Autonomic computing · doi:10.1145/2371536.2371562 · Cited by 225
“The task placement constraints specify additional scheduling constraints concerning the machine configurations, such as processor architecture of the physical machine [21].”
BackgroundResultSelf-citation
Metadata
- Authors
- Sharma, B., Chudnovsky, V., Hellerstein, J. L., Rifaat, R., & Das, C. R.
- Venue
- Proceedings of the 2nd ACM Symposium on Cloud Computing
- Publisher
- ACM
- Published
- 2011-10-26
- Pages
- 1-14
- OpenAlex
- W2028617807
- Cited by
- 184
- References
- 18
- FWCI
- 16.85
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.