The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Interpreting the Data: Parallel Analysis with Sawzall
Open accessAbstract
Very large data sets often have a flat but regular structure and span multiple disks and machines. Examples include telephone call records, network logs, and web document repositories. These large data sets are not amenable to study using traditional database techniques, if only because they can be too large to fit in a single relational database. On the other hand, many of the analyses done on them can be expressed using simple, easily distributed computations: filtering, aggregation, extraction of statistics, and so on. We present a system for automating such analyses. A filtering phase, in which a query is expressed using a new procedural programming language, emits data to an aggregation phase. Both phases are distributed over hundreds or even thousands of computers. The results are then collated and saved to a file. The design – including the separation into two phases, the form of the programming language, and the properties of the aggregators – exploits the parallelism inherent in having data and computation distributed across many machines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003) · ACM SIGOPS Operating Systems Review · doi:10.1145/1165389.945450 · Cited by 4,984
Clocksin, W. F., & Mellish, C. (1984) · Springer eBooks · doi:10.1007/978-3-642-55481-0 · Cited by 2,047
Distributed computing in practice: the Condor experience
Thain, D., Tannenbaum, T., & Livny, M. (2005) · Concurrency and Computation: Practice and Experience · doi:10.1002/cpe.938 · Cited by 1,886
Finding frequent items in data streams
Charikar, M., Chen, K., & Farach-Colton, M. (2004) · Theoretical Computer Science · doi:10.1016/s0304-3975(03)00400-6 · Cited by 1,278
Brook for GPUs: stream computing on graphics hardware
Buck, I., Foley, T., Horn, D., Sugerman, J., Fatahalian, K., Houston, M., & Hanrahan, P. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015800 · Cited by 1,155
Monitoring Streams — A New Class of Data Management Applications
Carney, D., Çetintemel, U., Cherniack, M., Convey, C., Lee, S.-D., Seidman, G., Tatbul, N., Zdonik, S., & Stonebraker, M. R. (2002) · Elsevier eBooks · doi:10.1016/b978-155860869-6/50027-5 · Cited by 842
Finding Frequent Items in Data Streams
Charikar, M., Chen, K., & Farach-Colton, M. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-45465-9_59 · Cited by 650
Space-efficient online computation of quantile summaries
Greenwald, M., & Khanna, S. (2001) · Proceedings of the 2001 ACM SIGMOD international conference on Management of data · doi:10.1145/375663.375670 · Cited by 451
Aho, A. V., Kernighan, B. W., & Weinberger, P. J. (1988) · CERN Document Server (European Organization for Nuclear Research) · Cited by 405
Cited by
Pregel: a system for large-scale graph processing
Malewicz, G., Austern, M. H., Bik, A. J. C., Dehnert, J. C., Horn, I., Leiser, N., & Czajkowski, G. (2010) · Proceedings of the 2010 ACM SIGMOD International Conference on Management of data · doi:10.1145/1807167.1807184 · Cited by 3,515
Bigtable: A Distributed Storage System for Structured Data
Chang, F., Dean, J., Ghemawat, S., Hsieh, W. C., Wallach, D. A., Burrows, M., Chandra, T., Fikes, A., & Gruber, R. E. (2008) · ACM Transactions on Computer Systems · doi:10.1145/1365815.1365816 · Cited by 3,460
Chen, M., Mao, S., & Liu, Y. (2014) · Mobile Networks and Applications · doi:10.1007/s11036-013-0489-0 · Cited by 2,913
The rise of big data on cloud computing: Review and open research issues
Hashem, I. A. T., Yaqoob, I., Anuar, N. B., Mokhtar, S., Gani, A., & Ullah Khan, S. (2015) · Information Systems · doi:10.1016/j.is.2014.07.006 · Cited by 2,656
Dryad: distributed data-parallel programs from sequential building blocks
Isard, M., Budiu, M., Yu, Y., Birrell, A., & Fetterly, D. (2007) · Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007 · doi:10.1145/1272996.1273005 · Cited by 2,432
Bigtable: a distributed storage system for structured data
Chang, F. W., Dean, J. B., Ghemawat, S., Hsieh, W. C., Wallach, D. A., Burrows, M., Chandra, T., Fikes, A., & Gruber, R. E. (2006) · doi:10.5555/1298455.1298475 · Cited by 1,928
Pig latin: a not-so-foreign language for data processing
Olston, C., Reed, B., Srivastava, U., Kumar, R., & Tomkins, A. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376726 · Cited by 1,730
The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines
Barroso, L. A., & Hölzle, U. (2009) · Synthesis lectures on computer architecture · doi:10.2200/s00193ed1v01y200905cac006 · Cited by 1,714
Improving MapReduce performance in heterogeneous environments
Zaharia, M., Konwinski, A., Joseph, A. D., Katz, R. H., & Stoica, I. G. (2008) · doi:10.5555/1855741.1855744 · Cited by 1,588
MapReduce: a flexible data processing tool
Dean, J., & Ghemawat, S. (2010) · Communications of the ACM · doi:10.1145/1629175.1629198 · Cited by 1,182
Toward Scalable Systems for Big Data Analytics: A Technology Tutorial
Hu, H., Wen, Y., Chua, T.-S., & Li, X. (2014) · IEEE Access · doi:10.1109/access.2014.2332453 · Cited by 1,099
SCOPE: easy and efficient parallel processing of massive data sets
Chaiken, R., Jenkins, B., Larson, P.-Å., Ramsey, B., Shakib, D., Weaver, S., & Zhou, J. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1454159.1454166 · Cited by 727
Map-reduce-merge: simplified relational data processing on large clusters
Yang, H.-C., Dasdan, A., Hsiao, R.-L., & Parker, D. S. (2007) · Proceedings of the 2007 ACM SIGMOD international conference on Management of data · doi:10.1145/1247480.1247602 · Cited by 715
Dremel: interactive analysis of web-scale datasets
Melnik, S., Gubarev, A., Long, J. J., Romer, G., Shivakumar, S., Tolton, M., & Vassilakis, T. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1920886 · Cited by 711
Yu, Y., Isard, M., Fetterly, D., Budiu, M., Erlingsson, Ú., Gunda, P. K., & Currey, J. (2008) · doi:10.5555/1855741.1855742 · Cited by 689
Condie, T., Conway, N., Alvaro, P., Hellerstein, J. M., Elmeleegy, K., & Sears, R. (2010) · doi:10.5555/1855711.1855732 · Cited by 673
PEGASUS: A Peta-Scale Graph Mining System Implementation and Observations
Kang, U., Tsourakakis, C. E., & Faloutsos, C. (2009) · 2009 Ninth IEEE International Conference on Data Mining · doi:10.1109/icdm.2009.14 · Cited by 628
Parallel data processing with MapReduce: a survey
Lee, K.-H., Lee, Y.-J., Choi, H., Chung, Y. D., & Moon, B. (2012) · ACM SIGMOD Record · doi:10.1145/2094114.2094118 · Cited by 558
DRAM errors in the wild: a large-scale field study
Schroeder, B., Pinheiro, E., & Weber, W.-D. (2009) · Proceedings of the eleventh international joint conference on Measurement and modeling of computer systems · doi:10.1145/1555349.1555372 · Cited by 557
“We solve this problem by using a parallel pre-processing step (implemented in Sawzall [18]), which runs on several hundred nodes simultaneously and performs basic data clean-up and filtering.”
MethodDryad: distributed data-parallel programs from sequential building blocks
Isard, M., Budiu, M., Yu, Y., Birrell, A., & Fetterly, D. (2007) · ACM SIGOPS Operating Systems Review · doi:10.1145/1272998.1273005 · Cited by 556
(2010) · Springer eBooks · doi:10.1007/978-1-4419-6524-0 · Cited by 519
Google’s MapReduce programming model — Revisited
Lämmel, R. (2008) · Science of Computer Programming · doi:10.1016/j.scico.2007.07.001 · Cited by 489
Barroso, L. A., Clidaras, J., & Hölzle, U. (2013) · Synthesis lectures on computer architecture · doi:10.2200/s00516ed2v01y201306cac024 · Cited by 489
Data-intensive text processing with MapReduce
Lin, J., & Dyer, C. (2009) · Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Tutorial Abstracts on - NAACL '09 · doi:10.3115/1620950.1620951 · Cited by 445
The Stratosphere platform for big data analytics
Alexandrov, A., Bergmann, R., Ewen, S., Freytag, J.-C., Hueske, F., Heise, A., Kao, O., Leich, M., Leser, U., Markl, V., et al. (2014) · The VLDB Journal · doi:10.1007/s00778-014-0357-y · Cited by 432
A Survey of Large Scale Data Management Approaches in Cloud Environments
Sakr, S., Liu, A., Batista, D. M., & Alomari, M. (2011) · IEEE Communications Surveys & Tutorials · doi:10.1109/surv.2011.032211.00087 · Cited by 424
Qian, L., Luo, Z., Du, Y., & Guo, L. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-10665-1_63 · Cited by 415
Building a high-level dataflow system on top of Map-Reduce: the Pig experience
Gates, A. F., Natkovich, O., Chopra, S., Kamath, P., Narayanamurthy, S. M., Olston, C., Reed, B., Srinivasan, S., & Srivastava, U. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687553.1687568 · Cited by 405
Swift: A language for distributed parallel scripting
Wilde, M., Hategan, M., Wozniak, J. M., Clifford, B., Katz, D. S., & Foster, I. (2011) · Parallel Computing · doi:10.1016/j.parco.2011.05.005 · Cited by 402
BigBench: towards an industry standard benchmark for big data analytics
Ghazal, A., Rabl, T., Hu, M., Raab, F., Poess, M., Crolotte, A., & Jacobsen, H.-A. (2013) · Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2463676.2463712 · Cited by 343
Coflow: a networking abstraction for cluster applications
Chowdhury, M., & Stoica, I. (2012) · Proceedings of the 11th ACM Workshop on Hot Topics in Networks · doi:10.1145/2390231.2390237 · Cited by 340
FlumeJava: easy, efficient data-parallel pipelines
Chambers, C., Raniwala, A., Perry, F., Adams, S., Henry, R. R., Bradshaw, R., & Weizenbaum, N. (2010) · Proceedings of the 31st ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/1806596.1806638 · Cited by 331
Distributed and cloud computing: from parallel processing to the Internet of things
(2012) · Choice Reviews Online · doi:10.5860/choice.49-5712 · Cited by 314
Hüttner, F. J., Fitzmaurice, C., Schwarzer, G., Seiler, C. M., Antes, G., Büchler, M. W., & Diener, M. K. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006053.pub6 · Cited by 313
Finding frequent items in data streams
Cormode, G., & Hadjieleftheriou, M. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1454159.1454225 · Cited by 304
Heule, S., Nunkesser, M., & Hall, A. (2013) · Proceedings of the 16th International Conference on Extending Database Technology · doi:10.1145/2452376.2452456 · Cited by 302
Exploiting Dynamic Resource Allocation for Efficient Parallel Data Processing in the Cloud
Warneke, D., & Kao, O. (2011) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2011.65 · Cited by 276
Boa: A language and infrastructure for analyzing ultra-large-scale software repositories
Dyer, R., Nguyen, H. A., Rajan, H., & Nguyen, T. N. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606588 · Cited by 255
Piccolo: building fast, distributed programs with partitioned tables
Power, R., & Li, J. (2010) · doi:10.5555/1924943.1924964 · Cited by 252
Measuring the user experience on a large scale: user-centered metrics for web applications
Rodden, K., Hutchinson, H., & Fu, X. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753687 · Cited by 233
Real-time big data processing framework: Challenges and solutions
(2015) · Applied Mathematics & Information Sciences · doi:10.12785/amis/090646 · Cited by 74
(2008) · Cited by 47
Metadata
- Authors
- Pike, R., Dorward, S., Griesemer, R., & Quinlan, S.
- Venue
- Scientific Programming
- Publisher
- Wiley
- Published
- 2005-01
- Volume
- 13
- Issue
- 4
- Pages
- 277-298
- ISSN
- 1058-9244 (ISSN-L)
- 1875-919X
- OpenAlex
- W2119714163
- Cited by
- 611
- References
- 15
- FWCI
- 24.47
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.