Patterns in the Chaos—A Study of Performance Variation and Predictability in Public IaaS Clouds
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Benchmarking the performance of public cloud providers is a common research topic. Previous work has already extensively evaluated the performance of different cloud platforms for different use cases, and under different constraints and experiment setups. In this article, we present a principled, large-scale literature review to collect and codify existing research regarding the predictability of performance in public Infrastructure-as-a-Service (IaaS) clouds. We formulate 15 hypotheses relating to the nature of performance variations in IaaS systems, to the factors of influence of performance variations, and how to compare different instance types. In a second step, we conduct extensive real-life experimentation on four cloud providers to empirically validate those hypotheses. We show that there are substantial differences between providers. Hardware heterogeneity is today less prevalent than reported in earlier research, while multitenancy has a dramatic impact on performance and predictability, but only for some cloud providers. We were unable to discover a clear impact of the time of the day or the day of the week on cloud performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Armbrust, M., Fox, A., Griffith, R., Joseph, A. D., Katz, R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A., Stoica, I., et al. (2010) · Communications of the ACM · doi:10.1145/1721654.1721672 · Cited by 9,464 · cited 1×
“In an Infrastructure-as-a-Service (IaaS) cloud [8], computing resources are acquired and released as a service, typically in the form of virtual machines with attached virtual disks [11].”
BackgroundBuyya, R., Yeo, C. S., Venugopal, S., Broberg, J., & Brandic, I. (2009) · Future Generation Computer Systems · doi:10.1016/j.future.2008.12.001 · Cited by 5,937 · cited 1×
“In an Infrastructure-as-a-Service (IaaS) cloud [8], computing resources are acquired and released as a service, typically in the form of virtual machines with attached virtual disks [11].”
BackgroundCloudCmp: comparing public cloud providers
Li, A., Yang, X., Kandula, S., & Zhang, M. (2010) · Proceedings of the 10th ACM SIGCOMM conference on Internet measurement · doi:10.1145/1879141.1879143 · Cited by 882 · cited 1×
“pports H1.1 based on various micro and application benchmarks, including CPU performance [OLoughlin and Gillam 2013; Akioka and Muraoka 2010], hard disk IO [Kossmann et al. 2010; Ghoshal et al. 2011; Li et al. 2010], memory [Atas¸ and Gungor 2014; Gillam et al. 2013], network latency [Barker and Shenoy 2010], or overa…”
BackgroundPerformance Analysis of Cloud Computing Services for Many-Tasks Scientific Computing
Iosup, A., Ostermann, S., Yigitbasi, M. N., Prodan, R., Fahringer, T., & Epema, D. H. J. (2011) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2011.66 · Cited by 867
Runtime measurements in the cloud: observing, analyzing, and reducing variance
Schad, J., Dittrich, J., & Quiané-Ruiz, J.-A. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1920902 · Cited by 670
(2004) · doi:10.5555/998675.999432 · Cited by 656
The Impact of Virtualization on Network Performance of Amazon EC2 Data Center
Wang, G., & Ng, T. S. E. (2010) · 2010 Proceedings IEEE INFOCOM · doi:10.1109/infcom.2010.5461931 · Cited by 655 · cited 1×
“For multi-tenancy dominated applications (e.g., disk I/O [16,39] and network throughput [45]), existing research has established that even multiple benchmarks taken in quick succession, within the timeframe of minutes or even seconds, can lead to substantially varying results.”
BackgroundPerformance Analysis of High Performance Computing Applications on the Amazon Web Services Cloud
Jackson, K. R., Ramakrishnan, L., Muriki, K., Canon, S., Cholia, S., Shalf, J., Wasserman, H. J., & Wright, N. J. (2010) · 2010 IEEE Second International Conference on Cloud Computing Technology and Science · doi:10.1109/cloudcom.2010.69 · Cited by 596
A Performance Study on the VM Startup Time in the Cloud
Mao, M., & Humphrey, M. (2012) · 2012 IEEE Fifth International Conference on Cloud Computing · doi:10.1109/cloud.2012.103 · Cited by 526 · cited 1×
“We seeded our review with 7 seminal publications, which we considered widely-known and representative of the field [17,21,22,28,32,37,38].”
MethodOn the Performance Variability of Production Cloud Services
Iosup, A., Yigitbasi, N., & Epema, D. (2011) · 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing · doi:10.1109/ccgrid.2011.22 · Cited by 328 · cited 1×
“Some authors have speculated that there may be additional temporal and geographical factors that influence cloud performance and predictability [Iosup et al. 2011b], in addition to hardware heterogeneity and multitenancy.”
Background
Cited by
Methodological Principles for Reproducible Performance Evaluation in Cloud Computing
Papadopoulos, A. V., Versluis, L., Bauer, A., Herbst, N., Kistowski, J. V., Ali-Eldin, A., Abad, C. L., Amaral, J. N., Tuma, P., & Iosup, A. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2927908 · Cited by 93
Metadata
- Authors
- Leitner, P., & Cito, J.
- Venue
- ACM Transactions on Internet Technology
- Published
- 2016-04-19
- Volume
- 16
- Issue
- 3
- Pages
- 1-23
- ISSN
- 1533-5399 (ISSN-L)
- 1557-6051
- OpenAlex
- W1651603302
- Cited by
- 225
- References
- 20
- FWCI
- 69.85
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Patterns in the Chaos—A Study of Performance Variation and Predictability in Public IaaS Clouds. https://app.cochat.ai/papers/10.1145/2885497
- The paper
- Leitner, P., & Cito, J. (2016). Patterns in the Chaos—A Study of Performance Variation and Predictability in Public IaaS Clouds. ACM Transactions on Internet Technology. https://doi.org/10.1145/2885497
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.