Autopilot: workload autoscaling at Google
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
In many public and private Cloud systems, users need to specify a limit for the amount of resources (CPU cores and RAM) to provision for their workloads. A job that exceeds its limits might be throttled or killed, resulting in delaying or dropping end-user requests, so human operators naturally err on the side of caution and request a larger limit than the job needs. At scale, this results in massive aggregate resource wastage.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
MapReduce: Simplified Data Processing on Large Cluster
Senior Software Developer, A. U., & Dayalan, M. (2018) · International Journal of Research and Engineering · doi:10.21276/ijre.2018.5.5.4 · Cited by 2,948
Apache Hadoop YARN: yet another resource negotiator
Vavilapalli, V. K., Murthy, A. C., Douglas, C., Agarwal, S., Konar, M., Evans, R., Graves, T., Lowe, J., Shah, H., Seth, S., et al. (2013) · Proceedings of the 4th annual Symposium on Cloud Computing · doi:10.1145/2523616.2523633 · Cited by 1,848 · cited 5×
“[30] shows a system oversubscribing resources in a YARN cluster.”
BackgroundInfluentialSLURM: Simple Linux Utility for Resource Management
Yoo, A. B., Jette, M. A., & Grondona, M. (2003) · Lecture Notes in Computer Science · doi:10.1007/10968987_3 · Cited by 1,535
Large-scale cluster management at Google with Borg
Verma, A., Pedrosa, L., Korupolu, M., Oppenheimer, D., Tune, E., & Wilkes, J. (2015) · Proceedings of the Tenth European Conference on Computer Systems · doi:10.1145/2741948.2741964 · Cited by 1,468 · cited 3×
“Borg [34], Omega [29] and Kubernetes [3] all require their users to set such limits when submitting jobs (Borg, Omega) or pods (Kubernetes).”
BackgroundMethodSelf-citationHeterogeneity and dynamicity of clouds at scale: Google trace analysis
Reiss, C., Tumanov, A., Ganger, G. R., Katz, R. H., & Kozuch, M. A. (2012) · Proceedings of the Third ACM Symposium on Cloud Computing · doi:10.1145/2391229.2391236 · Cited by 1,063
A Review of Auto-scaling Techniques for Elastic Applications in Cloud Environments
Lorido-Botran, T., Miguel-Alonso, J., & Lozano, J. A. (2014) · Journal of Grid Computing · doi:10.1007/s10723-014-9314-7 · Cited by 701
Omega: flexible, scalable schedulers for large compute clusters
Schwarzkopf, M., Konwinski, A., Abd-El-Malek, M., & Wilkes, J. (2013) · Proceedings of the 8th ACM European Conference on Computer Systems · doi:10.1145/2465351.2465386 · Cited by 650
Self-citationEmpirical prediction models for adaptive resource provisioning in the cloud
Islam, S., Keung, J., Lee, K., & Liu, A. (2012) · Future Generation Computer Systems · doi:10.1016/j.future.2011.05.027 · Cited by 631
Efficient Autoscaling in the Cloud Using Predictive Models for Workload Forecasting
Roy, N., Dubey, A., & Gokhale, A. (2011) · 2011 IEEE 4th International Conference on Cloud Computing · doi:10.1109/cloud.2011.42 · Cited by 591
Ernest: efficient performance prediction for large-scale advanced analytics
Venkataraman, S., Yang, Z., Franklin, M. J. M., Recht, B., & Stoica, I. G. (2016) · Networked Systems Design and Implementation · Cited by 559
PRESS: PRedictive Elastic ReSource Scaling for cloud systems
Gong, Z., Gu, X., & Wilkes, J. (2010) · 2010 International Conference on Network and Service Management · doi:10.1109/cnsm.2010.5691343 · Cited by 544
Self-citationBorg, Omega, and Kubernetes: Lessons learned from three container-management systems over a decade
Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. (2016) · Queue · doi:10.1145/2898442.2898444 · Cited by 443
Self-citationMulti-resource packing for cluster schedulers
Grandl, R., Ananthanarayanan, G., Kandula, S., Rao, S., & Akella, A. (2014) · Proceedings of the 2014 ACM conference on SIGCOMM · doi:10.1145/2619239.2626334 · Cited by 421
Cherrypick: adaptively unearthing the best cloud configurations for big data analytics
Alipourfard, O., Liu, H. H., Chen, J., Venkataraman, S., Yu, M., & Zhang, M. (2017) · Networked Systems Design and Implementation · Cited by 374
Tirmazi, M., Barker, A., Deng, N., Haque, M. E., Qin, Z. G., Hand, S., Harchol-Balter, M., & Wilkes, J. (2020) · Proceedings of the Fifteenth European Conference on Computer Systems · doi:10.1145/3342195.3387517 · Cited by 325
Self-citationCPI 2: CPU performance isolation for shared compute clusters
Zhang, X., Tune, E., Hagmann, R., Jnagal, R., Gokhale, V., & Wilkes, J. (2013) · Proceedings of the 8th ACM European Conference on Computer Systems · doi:10.1145/2465351.2465388 · Cited by 285
Self-citationQuasar: resource-efficient and QoS-aware cluster management
Delimitrou, C., & Kozyrakis, C. (2014) · ACM SIGPLAN Notices · doi:10.1145/2644865.2541941 · Cited by 231
Selecting the best VM across multiple public clouds: a data-driven performance modeling approach
Yadwadkar, N. J., Hariharan, B., Gonzalez, J. E., Smith, B. J., & Katz, R. H. (2017) · Symposium on Cloud Computing · doi:10.1145/3127479.3131614 · Cited by 195
Imbalance in the cloud: An analysis on Alibaba cluster trace
Lu, C., Ye, K., Xu, G., Xu, C.-Z., & Bai, T. (2017) · 2017 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2017.8258257 · Cited by 190
Breitgand, D., & Epstein, A. (2012) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infcom.2012.6195716 · Cited by 115
Janardhanan, D., & Barrett, E. (2017) · 2017 12th International Conference for Internet Technology and Secured Transactions (ICITST) · doi:10.23919/icitst.2017.8356346 · Cited by 106
Exploring Potential for Non-Disruptive Vertical Auto Scaling and Resource Estimation in Kubernetes
Rattihalli, G., Govindaraju, M., Lu, H., & Tiwari, D. (2019) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud.2019.00018 · Cited by 93
Self-Learning Cloud Controllers: Fuzzy Q-Learning for Knowledge Evolution
Jamshidi, P., Sharifloo, A. M., Pahl, C., Metzger, A., & Estrada, G. G. (2015) · International Conference on Cloud and Autonomic Computing (ICCAC) · doi:10.1109/iccac.2015.35 · Cited by 77
Hydra: a federated resource manager for data-center scale analytics
Curino, C., Krishnan, S., Karanasos, K., Rao, S., Fumarola, G. M., Huang, B., Chaliparambil, K., Suresh, A., Chen, Y., Heddaya, S., et al. (2019) · Networked Systems Design and Implementation · Cited by 74
An Analysis of Public Clouds Elasticity in the Execution of Scientific Applications: a Survey
Galante, G., Bona, L. C. E., Mury, A. R., Schulze, B. R., & Righi, R. D. R. (2016) · Journal of Grid Computing · doi:10.1007/s10723-016-9361-3 · Cited by 62
Performance modelling and verification of cloud-based auto-scaling policies
Evangelidis, A., Parker, D., & Rami Bahsoon, A. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2017.12.047 · Cited by 56
Adaptation in cloud resource configuration: a survey
Hummaida, A. R., Paton, N. W., & Sakellariou, R. (2016) · Journal of Cloud Computing Advances Systems and Applications · doi:10.1186/s13677-016-0057-9 · Cited by 52
A multitime‐steps‐ahead prediction approach for scheduling live migration in cloud data centers
Duggan, M., Shaw, R., Duggan, J., Howley, E., & Barrett, E. (2018) · Software Practice and Experience · doi:10.1002/spe.2635 · Cited by 52
ROSE: Cluster Resource Scheduling via Speculative Over-Subscription
Sun, X., Hu, C., Yang, R., Garraghan, P., Wo, T., Xu, J. J., Zhu, J., & Li, C. (2018) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2018.00096 · Cited by 49
2010 International Conference on Network and Service Management
(2010) · doi:10.1109/cnsm17656.2010 · Cited by 38
SLO-aware colocation of data center tasks based on instantaneous processor requirements
Janus, P., & Rzaḑca, K. (2017) · Symposium on Cloud Computing · doi:10.1145/3127479.3132244 · Cited by 32
Cited by
Tirmazi, M., Barker, A., Deng, N., Haque, M. E., Qin, Z. G., Hand, S., Harchol-Balter, M., & Wilkes, J. (2020) · Proceedings of the Fifteenth European Conference on Computer Systems · doi:10.1145/3342195.3387517 · Cited by 325
Self-citationA Survey of Autoscaling in Kubernetes
Tran, M.-N., Vu, D.-D., & Kim, Y. (2022) · 2022 Thirteenth International Conference on Ubiquitous and Future Networks (ICUFN) · doi:10.1109/icufn55119.2022.9829572 · Cited by 51
“In some works such as [4, 20], multiple deep neural networks are used at the same time with the best performer will be used for triggering autoscaling decisions.”
Background
Metadata
- Authors
- Rzadca, K., Findeisen, P., Swiderski, J., Zych, P., Broniek, P., Kusmierek, J., Nowak, P., Strack, B., Witusowski, P., Hand, S., & Wilkes, J.
- Venue
- Proceedings of the Fifteenth European Conference on Computer Systems
- Publisher
- ACM
- Published
- 2020-04-15
- Pages
- 1-16
- OpenAlex
- W3023238978
- Cited by
- 303
- References
- 32
- FWCI
- 26.59
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.