The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Interactive analytical processing in big data systems: a cross-industry study of MapReduce workloads
Abstract
Within the past few years, organizations in diverse industries have adopted MapReduce-based systems for large-scale data processing. Along with these new users, important new workloads have emerged which feature many small, short, and increasingly interactive jobs in addition to the large, long-running batch jobs for which MapReduce was originally designed. As interactive, large-scale query processing is a strength of the RDBMS community, it is important that lessons from that field be carried over and applied where possible in this new domain. However, these new workloads have not yet been described in the literature. We fill this gap with an empirical analysis of MapReduce traces from six separate business-critical deployments inside Facebook and at Cloudera customers in e-commerce, telecommunications, media, and retail. Our key contribution is a characterization of new MapReduce workloads which are driven in part by interactive analysis, and which make heavy use of query-like programming frameworks on top of MapReduce. These workloads display diverse behaviors which invalidate prior assumptions about MapReduce such as uniform data access, regular diurnal patterns, and prevalence of large jobs. A secondary contribution is a first step towards creating a TPC-like data processing benchmark for MapReduce.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
On the self-similar nature of Ethernet traffic (extended version)
Leland, W. E., Taqqu, M. S., Willinger, W., & Wilson, D. V. (1994) · IEEE/ACM Transactions on Networking · doi:10.1109/90.282603 · Cited by 5,233
Web caching and Zipf-like distributions: evidence and implications
Breslau, L., Cao, P., Fan, L., Phillips, G., & Shenker, S. (1999) · IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320) · doi:10.1109/infcom.1999.749260 · Cited by 3,467
A scalable, commodity data center network architecture
Al-Fares, M., Loukissas, A., & Vahdat, A. (2008) · ACM SIGCOMM Computer Communication Review · doi:10.1145/1402946.1402967 · Cited by 3,175
VL2: a scalable and flexible data center network
Greenberg, A., Hamilton, J. R., Jain, N., Kandula, S., Kim, C., Lahiri, P., Maltz, D. A., Patel, P., & Sengupta, S. (2009) · Proceedings of the ACM SIGCOMM 2009 conference on Data communication · doi:10.1145/1592568.1592576 · Cited by 2,111
Alizadeh, M., Greenberg, A., Maltz, D. A., Padhye, J., Patel, P., Prabhakar, B., Sengupta, S., & Sridharan, M. (2010) · Proceedings of the ACM SIGCOMM 2010 conference · doi:10.1145/1851182.1851192 · Cited by 1,877
Introduction to Machine Learning
(2019) · Series in Machine Perception and Artificial Intelligence · doi:10.1142/9789811201967_0001 · Cited by 1,639
(2011) · doi:10.5555/1972457.1972488 · Cited by 1,619
Hive: a warehousing solution over a map-reduce framework
Thusoo, A., Sarma, J. S., Jain, N., Shao, Z., Chakka, P., Anthony, S., Liu, H., Wyckoff, P., & Murthy, R. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687553.1687609 · Cited by 1,556
On the self-similar nature of Ethernet traffic
Leland, W. E., Taqqu, M. S., Willinger, W., & Wilson, D. V. (1993) · Conference proceedings on Communications architectures, protocols and applications · doi:10.1145/166237.166255 · Cited by 1,077
A comparison of approaches to large-scale data analysis
Pavlo, A., Paulson, E., Rasin, A., Abadi, D. J., DeWitt, D. J., Madden, S., & Stonebraker, M. (2009) · Proceedings of the 2009 ACM SIGMOD International Conference on Management of data · doi:10.1145/1559845.1559865 · Cited by 1,071
Quincy: fair scheduling for distributed computing clusters
Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., & Goldberg, A. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629601 · Cited by 856
Dremel: interactive analysis of web-scale datasets
Melnik, S., Gubarev, A., Long, J. J., Romer, G., Shivakumar, S., Tolton, M., & Vassilakis, T. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1920886 · Cited by 711
HaLoop: efficient iterative data processing on large clusters
Bu, Y., Howe, B., Balazinska, M., & Ernst, M. D. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1920881 · Cited by 693
Reining in the outliers in map-reduce clusters using Mantri
Ananthanarayanan, G., Kandula, S., Greenberg, A. G., Stoica, I. G., Lu, Y., Saha, B., & Harris, E. F. (2010) · Operating Systems Design and Implementation · doi:10.5555/1924943.1924962 · Cited by 636
Managing data transfers in computer clusters with orchestra
Chowdhury, M., Zaharia, M., Ma, J., Jordan, M. I., & Stoica, I. (2011) · Proceedings of the ACM SIGCOMM 2011 conference · doi:10.1145/2018436.2018448 · Cited by 543
End-to-end Internet packet dynamics
Paxson, V. (1997) · Proceedings of the ACM SIGCOMM '97 conference on Applications, technologies, architectures, and protocols for computer communication · doi:10.1145/263105.263155 · Cited by 542
A trace-driven analysis of the UNIX 4.2 BSD file system
Ousterhout, J. K., Da Costa, H., Harrison, D., Kunze, J. A., Kupfer, M., & Thompson, J. G. (1985) · ACM SIGOPS Operating Systems Review · doi:10.1145/323627.323631 · Cited by 495
The Case for Evaluating MapReduce Performance Using Workload Suites
Chen, Y., Ganapathi, A., Griffith, R., & Katz, R. (2011) · 2011 IEEE 19th Annual International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunication Systems · doi:10.1109/mascots.2011.12 · Cited by 428
Self-citationApache hadoop goes realtime at Facebook
Borthakur, D., Gray, J., Sarma, J. S., Muthukkaruppan, K., Spiegelberg, N., Kuang, H., Ranganathan, K., Molkov, D., Menon, A., Rash, S., et al. (2011) · Proceedings of the 2011 ACM SIGMOD International Conference on Management of data · doi:10.1145/1989323.1989438 · Cited by 425
Building a high-level dataflow system on top of Map-Reduce: the Pig experience
Gates, A. F., Natkovich, O., Chopra, S., Kamath, P., Narayanamurthy, S. M., Olston, C., Reed, B., Srinivasan, S., & Srivastava, U. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687553.1687568 · Cited by 405
Power management of online data-intensive services
Meisner, D., Sadler, C. M., Barroso, L. A., Weber, W.-D., & Wenisch, T. F. (2011) · Proceedings of the 38th annual international symposium on Computer architecture · doi:10.1145/2000064.2000103 · Cited by 404
Towards characterizing cloud backend workloads: insights from Google compute clusters
Mishra, A. K., Hellerstein, J. L., Cirne, W., & Das, C. R. (2010) · ACM SIGMETRICS Performance Evaluation Review · doi:10.1145/1773394.1773400 · Cited by 374
Profiling, what-if analysis, and cost-based optimization of MapReduce programs
Herodotou, H., & Babu, S. (2011) · Proceedings of the VLDB Endowment · doi:10.14778/3402707.3402746 · Cited by 368
Hadoop++: making a yellow elephant run like a cheetah (without it even noticing)
Dittrich, J., Quiané-Ruiz, J.-A., Jindal, A., Kargin, Y., Setty, V., & Schad, J. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1920908 · Cited by 364
Detecting BGP configuration faults with static analysis
Feamster, N., & Balakrishnan, H. (2005) · doi:10.5555/1251203.1251207 · Cited by 307
PACMan: coordinated memory caching for parallel jobs
Ananthanarayanan, G., Ghodsi, A., Wang, A., Borthakur, D., Kandula, S., Shenker, S. J., & Stoica, I. G. (2012) · Cited by 294
An analysis of data corruption in the storage stack
Bairavasundaram, L. N., Arpaci-Dusseau, A. C., Arpaci-Dusseau, R. H., Goodson, G. R., & Schroeder, B. (2008) · ACM Transactions on Storage · doi:10.1145/1416944.1416947 · Cited by 282
Statistics-driven workload modeling for the Cloud
Ganapathi, A., Chen, Y., Fox, A., Katz, R., & Patterson, D. (2010) · 2010 IEEE 26th International Conference on Data Engineering Workshops (ICDEW 2010) · doi:10.1109/icdew.2010.5452742 · Cited by 241
Self-citationEnergy management for MapReduce clusters
Lang, W., & Patel, J. M. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1920862 · Cited by 218
The case for persistent-connection HTTP
Mogul, J. C. (1995) · Proceedings of the conference on Applications, technologies, architectures, and protocols for computer communication · doi:10.1145/217382.217465 · Cited by 216
Quantitative Data Cleaning for Large Databases
Hellerstein, J. M. (2008) · Cited by 201
ParaTimer: a progress indicator for MapReduce DAGs
Morton, K., Balazinska, M., & Grossman, D. (2010) · Proceedings of the 2010 ACM SIGMOD International Conference on Management of data · doi:10.1145/1807167.1807223 · Cited by 182
Energy efficiency for large-scale MapReduce workloads with significant interactive analysis
Chen, Y., Alspaugh, S., Borthakur, D., & Katz, R. (2012) · Proceedings of the 7th ACM european conference on Computer Systems · doi:10.1145/2168836.2168842 · Cited by 175
Self-citationAutomatic optimization for MapReduce programs
Jahani, E., Cafarella, M. J., & Ré, C. (2011) · Proceedings of the VLDB Endowment · doi:10.14778/1978665.1978670 · Cited by 172
Sierra: practical power-proportionality for data center storage
Thereska, E., Donnelly, A., & Narayanan, D. (2011) · European Conference on Computer Systems · doi:10.1145/1966445.1966461 · Cited by 154
Cited by
The rise of big data on cloud computing: Review and open research issues
Hashem, I. A. T., Yaqoob, I., Anuar, N. B., Mokhtar, S., Gani, A., & Ullah Khan, S. (2015) · Information Systems · doi:10.1016/j.is.2014.07.006 · Cited by 2,656
Toward Scalable Systems for Big Data Analytics: A Technology Tutorial
Hu, H., Wen, Y., Chua, T.-S., & Li, X. (2014) · IEEE Access · doi:10.1109/access.2014.2332453 · Cited by 1,099
Big Data computing and clouds: trends and future directions
Assunção, M. D., Calheiros, R. N., Bianchi, S., Netto, M. A. S., & Buyya, R. (2015) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2014.08.003 · Cited by 901
“Previous work shows that there is an emerging class of MapReduce applications that feature small, short, and highly interactive jobs [73, 74].”
BackgroundStatistical Characterization of Business-Critical Workloads Hosted in Cloud Datacenters
Shen, S., Van Beek, V., & Iosup, A. (2015) · 2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing · doi:10.1109/ccgrid.2015.60 · Cited by 286
Recent Development in Big Data Analytics for Business Operations and Risk Management
Choi, T.-M., Chan, H. K., & Yue, X. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2507599 · Cited by 278
Metadata
- Authors
- Chen, Y., Alspaugh, S., & Katz, R.
- Venue
- Proceedings of the VLDB Endowment
- Published
- 2012-08
- Volume
- 5
- Issue
- 12
- Pages
- 1802-1813
- ISSN
- 2150-8097 (ISSN-L)
- OpenAlex
- W2104993419
- Cited by
- 469
- References
- 37
- FWCI
- 145.90
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.