The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
An Analysis of Traces from a Production MapReduce Cluster
Abstract
MapReduce is a programming paradigm for parallel processing that is increasingly being used for data-intensive applications in cloud computing environments. An understanding of the characteristics of workloads running in MapReduce environments benefits both the service providers in the cloud and users: the service provider can use this knowledge to make better scheduling decisions, while the user can learn what aspects of their jobs impact performance. This paper analyzes 10-months of MapReduce logs from the M45 supercomputing cluster which Yahoo! made freely available to select universities for academic research. We characterize resource utilization patterns, job patterns, and sources of failures. We use an instance-based learning technique that exploits temporal locality to predict job completion times from historical data and identify potential performance problems in our dataset.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003) · ACM SIGOPS Operating Systems Review · doi:10.1145/1165389.945450 · Cited by 4,984
Atkeson, C. G., Moore, A. W., & Schaal, S. (1997) · Artificial Intelligence Review · doi:10.1023/a:1006559212014 · Cited by 1,671
Ghemawat, S., Gobioff, H., & Leung, S.-T. A. (2003) · doi:10.1145/945449.945450 · Cited by 1,347
Improved Heterogeneous Distance Functions
Wilson, D. R., & Martinez, T. R. (1997) · Journal of Artificial Intelligence Research · doi:10.1613/jair.346 · Cited by 1,344
Quincy: fair scheduling for distributed computing clusters
Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., & Goldberg, A. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629601 · Cited by 856
Why Do Internet Services Fail, and What Can Be Done About It?
Oppenheimer, D., Ganapathi, A., & Patterson, D. A. (2002) · Cited by 607
A large-scale study of failures in high-performance computing systems
Schroeder, B., & Gibson, G. A. (2006) · International Conference on Dependable Systems and Networks (DSN'06) · doi:10.1109/dsn.2006.5 · Cited by 584
The workload on parallel supercomputers: modeling the characteristics of rigid jobs
Lublin, U., & Feitelson, D. G. (2003) · Journal of Parallel and Distributed Computing · doi:10.1016/s0743-7315(03)00108-4 · Cited by 441
An Analysis of Variance for Categorical Data
Light, R. J., & Margolin, B. H. (1971) · Journal of the American Statistical Association · doi:10.1080/01621459.1971.10482297 · Cited by 329
A census of Tandem system availability between 1985 and 1990
Gray, J. (1990) · IEEE Transactions on Reliability · doi:10.1109/24.58719 · Cited by 322
Collinearity and Least Squares Regression
Stewart, G. W. (1987) · Statistical Science · doi:10.1214/ss/1177013439 · Cited by 249
Workload Characteristics of a Multi-cluster Supercomputer
Li, H., Groep, D. L., & Wolters, L. (2005) · Lecture notes in computer science · doi:10.1007/11407522_10 · Cited by 137
Predictive application-performance modeling in a computational grid environment
Kapadia, N. H., Fortes, J. A. B., & Brodley, C. E. (2003) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1109/hpdc.1999.805281 · Cited by 128
SALSA: analyzing logs as state machines
Tan, J., Pan, X., Kavulya, S. P., Gandhi, R., & Narasimhan, P. (2008) · Cited by 95
Self-citationPrediction Services for Distributed Computing
Smith, W. W. (2007) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2007.370276 · Cited by 48
Cited by
Large-scale cluster management at Google with Borg
Verma, A., Pedrosa, L., Korupolu, M., Oppenheimer, D., Tune, E., & Wilkes, J. (2015) · Proceedings of the Tenth European Conference on Computer Systems · doi:10.1145/2741948.2741964 · Cited by 1,468
Heterogeneity and dynamicity of clouds at scale: Google trace analysis
Reiss, C., Tumanov, A., Ganger, G. R., Katz, R. H., & Kozuch, M. A. (2012) · Proceedings of the Third ACM Symposium on Cloud Computing · doi:10.1145/2391229.2391236 · Cited by 1,063
Omega: flexible, scalable schedulers for large compute clusters
Schwarzkopf, M., Konwinski, A., Abd-El-Malek, M., & Wilkes, J. (2013) · Proceedings of the 8th ACM European Conference on Computer Systems · doi:10.1145/2465351.2465386 · Cited by 650
Parallel data processing with MapReduce: a survey
Lee, K.-H., Lee, Y.-J., Choi, H., Chung, Y. D., & Moon, B. (2012) · ACM SIGMOD Record · doi:10.1145/2094114.2094118 · Cited by 558
ARIA: automatic resource inference and allocation for mapreduce environments
Verma, A., Cherkasova, L., & Campbell, R. H. (2011) · Proceedings of the 8th ACM international conference on Autonomic computing · doi:10.1145/1998582.1998637 · Cited by 410
Metadata
- Authors
- Kavulya, S., Tan, J., Gandhi, R., & Narasimhan, P.
- Venue
- 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing
- Publisher
- IEEE
- Published
- 2010
- Pages
- 94-103
- OpenAlex
- W2132353061
- Cited by
- 315
- References
- 16
- FWCI
- 21.36
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.