The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
ARIA: automatic resource inference and allocation for mapreduce environments
Abstract
MapReduce and Hadoop represent an economically compelling alternative for efficient large scale data processing and advanced analytics in the enterprise. A key challenge in shared MapReduce clusters is the ability to automatically tailor and control resource allocations to different applications for achieving their performance goals. Currently, there is no job scheduler for MapReduce environments that given a job completion deadline, could allocate the appropriate amount of resources to the job so that it meets the required Service Level Objective (SLO). In this work, we propose a framework, called ARIA, to address this problem. It comprises of three inter-related components. First, for a production job that is routinely executed on a new dataset, we build a job profile that compactly summarizes critical performance characteristics of the underlying application during the map and reduce stages. Second, we design a MapReduce performance model, that for a given job (with a known profile) and its SLO (soft deadline), estimates the amount of resources required for job completion within the deadline. Finally, we implement a novel SLO-based scheduler in Hadoop that determines job ordering and the amount of resources to allocate for meeting the job deadlines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
What is Twitter, a social network or a news media?
Kwak, H., Lee, C., Park, H., & Moon, S. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772751 · Cited by 6,652
Dryad: distributed data-parallel programs from sequential building blocks
Isard, M., Budiu, M., Yu, Y., Birrell, A., & Fetterly, D. (2007) · Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007 · doi:10.1145/1272996.1273005 · Cited by 2,432
Bounds for Certain Multiprocessing Anomalies
Graham, R. L. (1966) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1966.tb01709.x · Cited by 1,647
Improving MapReduce performance in heterogeneous environments
Zaharia, M., Konwinski, A., Joseph, A. D., Katz, R. H., & Stoica, I. G. (2008) · doi:10.5555/1855741.1855744 · Cited by 1,588
Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling
Zaharia, M., Borthakur, D., Sen Sarma, J., Elmeleegy, K., Shenker, S., & Stoica, I. (2010) · Proceedings of the 5th European conference on Computer systems · doi:10.1145/1755913.1755940 · Cited by 1,393
Quincy: fair scheduling for distributed computing clusters
Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., & Goldberg, A. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629601 · Cited by 856
Reining in the outliers in map-reduce clusters using Mantri
Ananthanarayanan, G., Kandula, S., Greenberg, A. G., Stoica, I. G., Lu, Y., Saha, B., & Harris, E. F. (2010) · Operating Systems Design and Implementation · doi:10.5555/1924943.1924962 · Cited by 636
Dryad: distributed data-parallel programs from sequential building blocks
Isard, M., Budiu, M., Yu, Y., Birrell, A., & Fetterly, D. (2007) · ACM SIGOPS Operating Systems Review · doi:10.1145/1272998.1273005 · Cited by 556
An Analysis of Traces from a Production MapReduce Cluster
Kavulya, S., Tan, J., Gandhi, R., & Narasimhan, P. (2010) · 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing · doi:10.1109/ccgrid.2010.112 · Cited by 315
Statistics-driven workload modeling for the Cloud
Ganapathi, A., Chen, Y., Fox, A., Katz, R., & Patterson, D. (2010) · 2010 IEEE 26th International Conference on Data Engineering Workshops (ICDEW 2010) · doi:10.1109/icdew.2010.5452742 · Cited by 241
Dynamic Proportional Share Scheduling in Hadoop
Sandholm, T., & Lai, K. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-16505-4_7 · Cited by 187
ParaTimer: a progress indicator for MapReduce DAGs
Morton, K., Balazinska, M., & Grossman, D. (2010) · Proceedings of the 2010 ACM SIGMOD International Conference on Management of data · doi:10.1145/1807167.1807223 · Cited by 182
FLEX: a slot allocation scheduling optimizer for MapReduce workloads
Wolf, J. L., Rajan, D., Hildrum, K. W., Khandekar, R., Kumar, V., Parekh, S. S., Wu, K., & Balmin, A. (2010) · ACM/IFIP/USENIX international conference on Middleware · doi:10.5555/2023718.2023720 · Cited by 151
Kahuna: Problem diagnosis for Mapreduce-based cloud computing environments
Tan, J., Pan, X., Marinelli, E., Kavulya, S. P., Gandhi, R., & Narasimhan, P. (2010) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms.2010.5488446 · Cited by 62
Cited by
(2017) · Networked Systems Design and Implementation · Cited by 329
Metadata
- Authors
- Verma, A., Cherkasova, L., & Campbell, R. H.
- Venue
- Proceedings of the 8th ACM international conference on Autonomic computing
- Publisher
- ACM
- Published
- 2011-06-14
- Pages
- 235-244
- OpenAlex
- W2002472616
- Cited by
- 410
- References
- 15
- FWCI
- 29.32
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.