Patterns in the Chaos—A Study of Performance Variation and Predictability in Public IaaS Clouds
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Benchmarking the performance of public cloud providers is a common research topic. Previous work has already extensively evaluated the performance of different cloud platforms for different use cases, and under different constraints and experiment setups. In this article, we present a principled, large-scale literature review to collect and codify existing research regarding the predictability of performance in public Infrastructure-as-a-Service (IaaS) clouds. We formulate 15 hypotheses relating to the nature of performance variations in IaaS systems, to the factors of influence of performance variations, and how to compare different instance types. In a second step, we conduct extensive real-life experimentation on four cloud providers to empirically validate those hypotheses. We show that there are substantial differences between providers. Hardware heterogeneity is today less prevalent than reported in earlier research, while multitenancy has a dramatic impact on performance and predictability, but only for some cloud providers. We were unable to discover a clear impact of the time of the day or the day of the week on cloud performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Armbrust, M., Fox, A., Griffith, R., Joseph, A. D., Katz, R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A., Stoica, I., et al. (2010) · Communications of the ACM · doi:10.1145/1721654.1721672 · Cited by 9,464 · cited 1×
“In an Infrastructure-as-a-Service (IaaS) cloud [8], computing resources are acquired and released as a service, typically in the form of virtual machines with attached virtual disks [11].”
BackgroundBuyya, R., Yeo, C. S., Venugopal, S., Broberg, J., & Brandic, I. (2009) · Future Generation Computer Systems · doi:10.1016/j.future.2008.12.001 · Cited by 5,937 · cited 1×
“In an Infrastructure-as-a-Service (IaaS) cloud [8], computing resources are acquired and released as a service, typically in the form of virtual machines with attached virtual disks [11].”
BackgroundCloudCmp: comparing public cloud providers
Li, A., Yang, X., Kandula, S., & Zhang, M. (2010) · Proceedings of the 10th ACM SIGCOMM conference on Internet measurement · doi:10.1145/1879141.1879143 · Cited by 882 · cited 1×
“pports H1.1 based on various micro and application benchmarks, including CPU performance [OLoughlin and Gillam 2013; Akioka and Muraoka 2010], hard disk IO [Kossmann et al. 2010; Ghoshal et al. 2011; Li et al. 2010], memory [Atas¸ and Gungor 2014; Gillam et al. 2013], network latency [Barker and Shenoy 2010], or overa…”
BackgroundPerformance Analysis of Cloud Computing Services for Many-Tasks Scientific Computing
Iosup, A., Ostermann, S., Yigitbasi, M. N., Prodan, R., Fahringer, T., & Epema, D. H. J. (2011) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2011.66 · Cited by 867
Runtime measurements in the cloud: observing, analyzing, and reducing variance
Schad, J., Dittrich, J., & Quiané-Ruiz, J.-A. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1920902 · Cited by 670
(2004) · doi:10.5555/998675.999432 · Cited by 656
The Impact of Virtualization on Network Performance of Amazon EC2 Data Center
Wang, G., & Ng, T. S. E. (2010) · 2010 Proceedings IEEE INFOCOM · doi:10.1109/infcom.2010.5461931 · Cited by 655 · cited 1×
“For multi-tenancy dominated applications (e.g., disk I/O [16,39] and network throughput [45]), existing research has established that even multiple benchmarks taken in quick succession, within the timeframe of minutes or even seconds, can lead to substantially varying results.”
BackgroundPerformance Analysis of High Performance Computing Applications on the Amazon Web Services Cloud
Jackson, K. R., Ramakrishnan, L., Muriki, K., Canon, S., Cholia, S., Shalf, J., Wasserman, H. J., & Wright, N. J. (2010) · 2010 IEEE Second International Conference on Cloud Computing Technology and Science · doi:10.1109/cloudcom.2010.69 · Cited by 596
A Performance Study on the VM Startup Time in the Cloud
Mao, M., & Humphrey, M. (2012) · 2012 IEEE Fifth International Conference on Cloud Computing · doi:10.1109/cloud.2012.103 · Cited by 526 · cited 1×
“We seeded our review with 7 seminal publications, which we considered widely-known and representative of the field [17,21,22,28,32,37,38].”
MethodOn the Performance Variability of Production Cloud Services
Iosup, A., Yigitbasi, N., & Epema, D. (2011) · 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing · doi:10.1109/ccgrid.2011.22 · Cited by 328 · cited 1×
“Some authors have speculated that there may be additional temporal and geographical factors that influence cloud performance and predictability [Iosup et al. 2011b], in addition to hardware heterogeneity and multitenancy.”
BackgroundEmpirical evaluation of latency-sensitive application performance in the cloud
Barker, S., & Shenoy, P. J. (2010) · ACM Multimedia Systems Conference (MMSys) · doi:10.1145/1730836.1730842 · Cited by 219 · cited 1×
“Contrary, the performance of multi-tenancy dominated applications or benchmarks, such as those that primarily depend on disk I/O or networking bandwidth, varies primarily due to the behavior of other co-located tenants [9,22].”
BackgroundMore for your money: exploiting performance heterogeneity in public clouds
Farley, B., Juels, A., Varadarajan, V., Ristenpart, T., Bowers, K. D., & Swift, M. M. (2012) · Proceedings of the Third ACM Symposium on Cloud Computing · doi:10.1145/2391229.2391249 · Cited by 169 · cited 3×
“This has been studied (and exploited) extensively in [16,28,35,36].”
BackgroundMigrate or not? exploiting dynamic task migration in mobile cloud computing systems
Gkatzikis, L., & Koutsopoulos, I. (2013) · IEEE Wireless Communications · doi:10.1109/mwc.2013.6549280 · Cited by 136 · cited 1×
“not CPU cores) are shared among all instances on a machine. This can lead to the “noisy neighbour” problem, when one instance experiences a slowdown due to the behavior of a co-located other tenant [Gkatzikis and Koutsopoulos 2013]. 3. RELATED WORK Our study takes inspiration from the efforts of Li et al., who have be…”
MethodOn a Catalogue of Metrics for Evaluating Commercial Cloud Services
Li, Z., O’Brien, L., Zhang, H., & Cai, R. (2012) · IEEE/ACM International Conference on Grid Computing (GRID) · doi:10.1109/grid.2012.15 · Cited by 133 · cited 1×
“Their work has led to a methodology for evaluating providers [29], to a catalogue of widely-used metrics [30], and to a conceptual model of IaaS benchmarks, capturing the various different dimensions that benchmark designers need to take into account [31].”
MethodInfrastructure as a Service and Cloud Technologies
Serrano, N., Gallardo, G., & Hernantes, J. (2015) · IEEE Software · doi:10.1109/ms.2015.43 · Cited by 97 · cited 1×
“. EC2 has been chosen due to both, its popularity in existing benchmarking research and its high industrial relevance. Similarly, Azure was chosen because it is seen as another current market leader [Serrano et al. 2015]. GCE, on the other hand, was interesting for us as the service went out of beta only 1 month prior…”
BackgroundCloud Work Bench -- Infrastructure-as-Code Based Cloud Benchmarking
Scheuner, J., Leitner, P., Cito, J., et al. (2014) · IEEE International Conference on Cloud Computing Technology and Science · doi:10.1109/cloudcom.2014.98 · Cited by 41 · cited 4×
“Previous research has proposed multiple approaches to achieve this, including Expertus [23], CloudBench [40], CloudCrawler [13], and Cloud Workbench [39].”
BackgroundMethodResultInfluentialSelf-citationSnow White Clouds and the Seven Dwarfs
Phillips, S., Engen, V., & Papay, J. (2011) · doi:10.1109/cloudcom.2011.114 · Cited by 39 · cited 1×
“We seeded our review with 7 seminal publications, which we considered widely-known and representative of the field [17,21,22,28,32,37,38].”
MethodCloud WorkBench: Benchmarking IaaS Providers based on Infrastructure-as-Code
Scheuner, J., Cito, J., Leitner, P., & Gall, H. C. (2015) · International World Wide Web Conference (WWW) · doi:10.1145/2740908.2742833 · Cited by 32 · cited 5×
“Using Cloud Workbench, we scheduled this procedure in 58 configurations 6 times per day over a period of approximately one month during July/August 2014.”
MethodInfluentialSelf-citationCEEM: A Practical Methodology for Cloud Services Evaluation
Li, Z., O’Brien, L., & Zhang, H. (2013) · IEEE World Congress on Services (SERVICES) · doi:10.1109/services.2013.73 · Cited by 23 · cited 1×
“Their work has led to a methodology for evaluating providers [29], to a catalogue of widely-used metrics [30], and to a conceptual model of IaaS benchmarks, capturing the various different dimensions that benchmark designers need to take into account [31].”
MethodZabolotnyi, R., Leitner, P., Hummer, W., & Dustdar, S. (2015) · ACM Transactions on Internet Technology · doi:10.1145/2792980 · Cited by 12 · cited 1×
“Additionally, we defined two application benchmarks, measuring queries per minute on a MySQL database (OLTP) and the Git checkout and Java compilation time of the open source project jCloudScale [Leitner et al. 2012; Zabolotnyi et al. 2015] (Java).”
MethodSelf-citation
Cited by
Methodological Principles for Reproducible Performance Evaluation in Cloud Computing
Papadopoulos, A. V., Versluis, L., Bauer, A., Herbst, N., Kistowski, J. V., Ali-Eldin, A., Abad, C. L., Amaral, J. N., Tuma, P., & Iosup, A. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2927908 · Cited by 93
Metadata
- Authors
- Leitner, P., & Cito, J.
- Venue
- ACM Transactions on Internet Technology
- Published
- 2016-04-19
- Volume
- 16
- Issue
- 3
- Pages
- 1-23
- ISSN
- 1533-5399 (ISSN-L)
- 1557-6051
- OpenAlex
- W1651603302
- Cited by
- 225
- References
- 20
- FWCI
- 69.85
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.