Runtime measurements in the cloud: observing, analyzing, and reducing variance
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
One of the main reasons why cloud computing has gained so much popularity is due to its ease of use and its ability to scale computing resources on demand. As a result, users can now rent computing nodes on large commercial clusters through several vendors, such as Amazon and rackspace. However, despite the attention paid by Cloud providers, performance unpredictability is a major issue in Cloud computing for (1) database researchers performing wall clock experiments, and (2) database applications providing service-level agreements. In this paper, we carry out a study of the performance variance of the most widely used Cloud infrastructure (Amazon EC2) from different perspectives. We use established microbenchmarks to measure performance variance in CPU, I/O, and network. And, we use a multi-node MapReduce application to quantify the impact on real dataintensive applications. We collected data for an entire month and compare it with the results obtained on a local cluster. Our results show that EC2 performance varies a lot and often falls into two bands having a large performance gap in-between --- which is somewhat surprising. We observe in our experiments that these two bands correspond to the different virtual system types provided by Amazon. Moreover, we analyze results considering different availability zones, points in time, and locations. This analysis indicates that, among others, the choice of availability zone also influences the performance variability. A major conclusion of our work is that the variance on EC2 is currently so high that wall clock experiments may only be performed with considerable care. To this end, we provide some hints to users.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
Above the Clouds: A Berkeley View of Cloud Computing
Armbrust, M., Fox, A., Griffith, R., Joseph, A. D., Katz, R. H., Konwinski, A., Lee, G., Patterson, D. A., Rabkin, A., Stoica, I. G., et al. (2009) · UC Berkeley · Cited by 6,274
Benchmarking cloud serving systems with YCSB
Cooper, B. F., Silberstein, A., Tam, E., Ramakrishnan, R., & Sears, R. (2010) · Proceedings of the 1st ACM symposium on Cloud computing · doi:10.1145/1807128.1807152 · Cited by 3,716
(1992) · Choice Reviews Online · doi:10.5860/choice.29-3347 · Cited by 2,095
Hey, you, get off of my cloud: exploring information leakage in third-party compute clouds
Ristenpart, T., Tromer, E., Shacham, H., & Savage, S. (2009) · Proceedings of the 16th ACM conference on Computer and communications security · doi:10.1145/1653662.1653687 · Cited by 2,053
A comparison of approaches to large-scale data analysis
Pavlo, A., Paulson, E., Rasin, A., Abadi, D. J., DeWitt, D. J., Madden, S., & Stonebraker, M. (2009) · Proceedings of the 2009 ACM SIGMOD International Conference on Management of data · doi:10.1145/1559845.1559865 · Cited by 1,071
A Performance Analysis of EC2 Cloud Computing Services for Scientific Computing
Ostermann, S., Iosup, A., Yigitbasi, N., Prodan, R., Fahringer, T., & Epema, D. H. J. (2010) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-642-12636-9_9 · Cited by 527
What's inside the Cloud? An architectural map of the Cloud landscape
Lenk, A., Klems, M., Nimis, J., Tai, S., & Sandholm, T. (2009) · 2009 ICSE Workshop on Software Engineering Challenges of Cloud Computing · doi:10.1109/cloud.2009.5071529 · Cited by 459
An Evaluation of Amazon's Grid Computing Services: EC2, S3, and SQS
Garfinkel, S. (2007) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 338
Can cloud computing reach the top500?
Napper, J., & Bientinesi, P. (2009) · Proceedings of the combined workshops on UnConventional high performance computing workshop plus memory access workshop · doi:10.1145/1531666.1531671 · Cited by 159
Cited by
A framework for ranking of cloud computing services
Garg, S. K., Versteeg, S., & Buyya, R. (2013) · Future Generation Computer Systems · doi:10.1016/j.future.2012.06.006 · Cited by 887
Towards predictable datacenter networks
Ballani, H., Costa, P., Karagiannis, T., & Rowstron, A. (2011) · Proceedings of the ACM SIGCOMM 2011 conference · doi:10.1145/2018436.2018465 · Cited by 750
Deadline Based Resource Provisioningand Scheduling Algorithm for Scientific Workflows on Clouds
Rodriguez, M. A., & Buyya, R. (2014) · IEEE Transactions on Cloud Computing · doi:10.1109/tcc.2014.2314655 · Cited by 686
Aceto, G., Botta, A., de Donato, W., & Pescapè, A. (2013) · Computer Networks · doi:10.1016/j.comnet.2013.04.001 · Cited by 554
Resource Management in Clouds: Survey and Research Challenges
Jennings, B., & Stadler, R. (2015) · Journal of Network and Systems Management · doi:10.1007/s10922-014-9307-7 · Cited by 548
OLTP-Bench: an extensible testbed for benchmarking relational databases
Difallah, D. E., Pavlo, A., Curino, C., & Cudre-Mauroux, P. (2013) · Proceedings of the VLDB Endowment · doi:10.14778/2732240.2732246 · Cited by 392
Hadoop++: making a yellow elephant run like a cheetah (without it even noticing)
Dittrich, J., Quiané-Ruiz, J.-A., Jindal, A., Kargin, Y., Setty, V., & Schad, J. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1920908 · Cited by 364
Self-citationXu, F., Liu, F., Jin, H., & Vasilakos, A. V. (2014) · Proceedings of the IEEE · doi:10.1109/jproc.2013.2287711 · Cited by 285
Seer: Leveraging Big Data to Navigate the Complexity of Performance Debugging in Cloud Microservices
Gan, Y., Zhang, Y., Hu, K., Cheng, D., He, Y., Pancholi, M., & Delimitrou, C. (2019) · Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3297858.3304004 · Cited by 269
Quality-of-service in cloud computing: modeling techniques and their applications
Ardagna, D., Casale, G., Ciavotta, M., Pérez, J. F., & Wang, W. (2014) · Journal of Internet Services and Applications · doi:10.1186/s13174-014-0011-3 · Cited by 267
Workflow scheduling in cloud: a survey
Wu, F., Wu, Q., & Tan, Y. (2015) · The Journal of Supercomputing · doi:10.1007/s11227-015-1438-4 · Cited by 260
Cloud Computing Features, Issues, and Challenges: A Big Picture
Puthal, D., Sahoo, B. P. S., Mishra, S., & Swain, S. (2015) · 2015 International Conference on Computational Intelligence and Networks · doi:10.1109/cine.2015.31 · Cited by 256
Methodological Principles for Reproducible Performance Evaluation in Cloud Computing
Papadopoulos, A. V., Versluis, L., Bauer, A., Herbst, N., Kistowski, J. V., Ali-Eldin, A., Abad, C. L., Amaral, J. N., Tuma, P., & Iosup, A. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2927908 · Cited by 93
Rimba, P., Tran, A. B., Weber, I., Staples, M., Ponomarev, A., & Xu, X. (2020) · Information Systems Frontiers · doi:10.1007/s10796-018-9876-1 · Cited by 68
Taming unbalanced training workloads in deep learning with partial collective operations
Li, S., Ben-Nun, T., Girolamo, S. D., Alistarh, D., & Hoefler, T. (2020) · Proceedings of the 25th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3332466.3374528 · Cited by 57
“finishthecomputationlaterthanothers, allprocesseswillwaitforthelastoneattheblocking allreduce function. Load imbalance can be caused by thesystemitself,forexample,whentrainingonmultitenantcloudsystems[30,31,46]orbysystemornetwork noise[26,27]inhigh-performancemachines.Asecond, andmoreprominentcauseofimbalanceisinheren…”
BackgroundInfluential
Metadata
- Authors
- Schad, J., Dittrich, J., & Quiané-Ruiz, J.-A.
- Venue
- Proceedings of the VLDB Endowment
- Published
- 2010-09
- Volume
- 3
- Issue
- 1-2
- Pages
- 460-471
- ISSN
- 2150-8097 (ISSN-L)
- OpenAlex
- W2091765165
- Cited by
- 670
- References
- 10
- FWCI
- 157.22
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.