The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Exploiting Dynamic Resource Allocation for Efficient Parallel Data Processing in the Cloud
Abstract
In recent years ad hoc parallel data processing has emerged to be one of the killer applications for Infrastructure-as-a-Service (IaaS) clouds. Major Cloud computing companies have started to integrate frameworks for parallel data processing in their product portfolio, making it easy for customers to access these services and to deploy their programs. However, the processing frameworks which are currently used have been designed for static, homogeneous cluster setups and disregard the particular nature of a cloud. Consequently, the allocated compute resources may be inadequate for big parts of the submitted job and unnecessarily increase processing time and cost. In this paper, we discuss the opportunities and challenges for efficient parallel data processing in clouds and present our research project Nephele. Nephele is the first data processing framework to explicitly exploit the dynamic resource allocation offered by today's IaaS clouds for both, task scheduling and execution. Particular tasks of a processing job can be assigned to different types of virtual machines which are automatically instantiated and terminated during the job execution. Based on this new framework, we perform extended evaluations of MapReduce-inspired processing jobs on an IaaS cloud system and compare the results to the popular data processing framework Hadoop.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
Globus: a Metacomputing Infrastructure Toolkit
Foster, I., & Kesselman, C. (1997) · The International Journal of Supercomputer Applications and High Performance Computing · doi:10.1177/109434209701100205 · Cited by 3,133
White, T. (2009) · Cited by 2,800
Dryad: distributed data-parallel programs from sequential building blocks
Isard, M., Budiu, M., Yu, Y., Birrell, A., & Fetterly, D. (2007) · Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007 · doi:10.1145/1272996.1273005 · Cited by 2,432
Pig latin: a not-so-foreign language for data processing
Olston, C., Reed, B., Srivastava, U., Kumar, R., & Tomkins, A. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376726 · Cited by 1,730
kvm : the Linux Virtual Machine Monitor
Kivity, A. (2007) · Cited by 1,267
Pegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems
Deelman, E., Singh, G., Su, M.-H., Blythe, J., Gil, Y., Kesselman, C., Mehta, G., Vahi, K., Berriman, G. B., Good, J., et al. (2005) · Scientific Programming · doi:10.1155/2005/128026 · Cited by 1,222
SCOPE: easy and efficient parallel processing of massive data sets
Chaiken, R., Jenkins, B., Larson, P.-Å., Ramsey, B., Shakib, D., Weaver, S., & Zhou, J. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1454159.1454166 · Cited by 727
Map-reduce-merge: simplified relational data processing on large clusters
Yang, H.-C., Dasdan, A., Hsiao, R.-L., & Parker, D. S. (2007) · Proceedings of the 2007 ACM SIGMOD international conference on Management of data · doi:10.1145/1247480.1247602 · Cited by 715
Interpreting the Data: Parallel Analysis with Sawzall
Pike, R., Dorward, S., Griesemer, R., & Quinlan, S. (2005) · Scientific Programming · doi:10.1155/2005/962135 · Cited by 611
Cited by
Resource Management in Clouds: Survey and Research Challenges
Jennings, B., & Stadler, R. (2015) · Journal of Network and Systems Management · doi:10.1007/s10922-014-9307-7 · Cited by 548
A Parallel Random Forest Algorithm for Big Data in a Spark Cloud Computing Environment
Chen, J., Li, K., Tang, Z., Bilal, K., Yu, S., Weng, C., & Li, K. (2017) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2016.2603511 · Cited by 435
The Stratosphere platform for big data analytics
Alexandrov, A., Bergmann, R., Ewen, S., Freytag, J.-C., Hueske, F., Heise, A., Kao, O., Leich, M., Leser, U., Markl, V., et al. (2014) · The VLDB Journal · doi:10.1007/s00778-014-0357-y · Cited by 432
Self-citationVerma, S., Kawamoto, Y., Fadlullah, Z. M., Nishiyama, H., & Kato, N. (2017) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2017.2694469 · Cited by 422
A Descriptive Literature Review and Classification of Cloud Computing Research
Yang, H., & Tate, M. (2012) · Communications of the Association for Information Systems · doi:10.17705/1cais.03102 · Cited by 330
Nguyen, G. N., Viet, N. H. L., Elhoseny, M., Shankar, K., Gupta, B. B., & El-Latif, A. A. A. (2021) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2021.03.011 · Cited by 306
Tsai, J.-T., Fang, J.-C., & Chou, J.-H. (2013) · Computers & Operations Research · doi:10.1016/j.cor.2013.06.012 · Cited by 272
Cloud-integrated cyber-physical systems for complex industrial applications
Shu, Z., Wan, J., Zhang, D., & Li, D. (2016) · Mobile Networks and Applications · doi:10.1007/s11036-015-0664-6 · Cited by 144
Metadata
- Authors
- Warneke, D., & Kao, O.
- Venue
- IEEE Transactions on Parallel and Distributed Systems
- Published
- 2011-06
- Volume
- 22
- Issue
- 6
- Pages
- 985-997
- ISSN
- 1045-9219 (ISSN-L)
- OpenAlex
- W2006140865
- Cited by
- 276
- References
- 18
- FWCI
- 108.20
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.