The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
MapReduce: Simplified Data Processing on Large Cluster
Abstract
MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper. Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program's execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system. Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google's clusters every day.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003) · ACM SIGOPS Operating Systems Review · doi:10.1145/1165389.945450 · Cited by 4,984
A Bridging Model for Parallel Computation
Valiant, L. G. (1990) · ACM eBooks · doi:10.1145/3749740.3749753 · Cited by 3,664
Efficient dispersal of information for security, load balancing, and fault tolerance
Rabin, M. O. (1989) · Journal of the ACM · doi:10.1145/62044.62050 · Cited by 1,365
Ladner, R. E., & Fischer, M. J. (1980) · Journal of the ACM · doi:10.1145/322217.322232 · Cited by 1,281
Web search for a planet: the google cluster architecture
Barroso, L. A., Dean, J., & Holzle, U. (2003) · IEEE Micro · doi:10.1109/mm.2003.1196112 · Cited by 1,025
Scans as primitive parallel operations
Blelloch, G. E. (1989) · IEEE Transactions on Computers · doi:10.1109/12.42122 · Cited by 538
Cluster-based scalable network services
Fox, A., Gribble, S. D., Chawathe, Y., Brewer, E. A., & Gauthier, P. (1997) · Proceedings of the sixteenth ACM symposium on Operating systems principles · doi:10.1145/268998.266662 · Cited by 522
Charlotte: Metacomputing on the Web
Baratloo, A., Karaul, M., Kedem, Z. M., & Wijckoff, P. (1999) · Future Generation Computer Systems · doi:10.1016/s0167-739x(99)00009-6 · Cited by 241
Active disks for large-scale data processing
Riedel, E., Faloutsos, C., Gibson, G. A., & Nagle, D. (2001) · Computer · doi:10.1109/2.928624 · Cited by 224
Cluster I/O with River: making the fast case common
Arpaci-Dusseau, R. H., Anderson, E., Treuhaft, N., Culler, D. E., Hellerstein, J. M., Patterson, D., & Yelick, K. (1999) · Proceedings of the sixth workshop on I/O in parallel and distributed systems · doi:10.1145/301816.301823 · Cited by 176
Cited by
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Buyya, R., Yeo, C. S., Venugopal, S., Broberg, J., & Brandic, I. (2009) · Future Generation Computer Systems · doi:10.1016/j.future.2008.12.001 · Cited by 5,937
The Hadoop Distributed File System
Shvachko, K., Kuang, H., Radia, S., & Chansler, R. (2010) · 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST) · doi:10.1109/msst.2010.5496972 · Cited by 4,834
Apache Spark: A unified engine for big data processing
Zaharia, M., Xin, R. S., Wendell, P., Das, T., Armbrust, M., Dave, A., Meng, X., Rosen, J., Venkataraman, S., Franklin, M. J., et al. (2016) · Communications of the ACM · doi:10.1145/2934664 · Cited by 2,757
Bigtable: a distributed storage system for structured data
Chang, F. W., Dean, J. B., Ghemawat, S., Hsieh, W. C., Wallach, D. A., Burrows, M., Chandra, T., Fikes, A., & Gruber, R. E. (2006) · doi:10.5555/1298455.1298475 · Cited by 1,928
Map-Reduce for Machine Learning on Multicore
Chu, C.-T., Kim, S. K., Lin, Y.-A., Yu, Y., Bradski, G., Ng, A. Y., & Olukotun, K. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0040 · Cited by 1,229
MapReduce: a flexible data processing tool
Dean, J., & Ghemawat, S. (2010) · Communications of the ACM · doi:10.1145/1629175.1629198 · Cited by 1,182
The nature of data center traffic: measurements & analysis
Kandula, S., Sengupta, S., Greenberg, A., Patel, P., & Chaiken, R. (2009) · Proceedings of the 9th ACM SIGCOMM conference on Internet measurement · doi:10.1145/1644893.1644918 · Cited by 1,030
Machine learning with big data: challenges and approaches.
L'Heureux, A., Grolinger, K., Elyamany, H. F., & Capretz, M. A. M. (2017) · IEEE Access · doi:10.1109/access.2017.2696365 · Cited by 1,023
Evaluating MapReduce for Multi-core and Multiprocessor Systems
Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., & Kozyrakis, C. (2007) · 2007 IEEE 13th International Symposium on High Performance Computer Architecture · doi:10.1109/hpca.2007.346181 · Cited by 966
Social influence analysis in large-scale networks
Tang, J., Sun, J., Wang, C., & Yang, Z. (2009) · Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1557019.1557108 · Cited by 927
Kambatla, K., Kollias, G., Kumar, V., & Grama, A. (2014) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2014.01.003 · Cited by 819
Akidau, T., Bradshaw, R., Chambers, C., Chernyak, S., Fernández-Moctezuma, R. J., Lax, R., McVeety, S., Mills, D., Perry, F., Schmidt, E., et al. (2015) · Proceedings of the VLDB Endowment · doi:10.14778/2824032.2824076 · Cited by 750
Towards predictable datacenter networks
Ballani, H., Costa, P., Karagiannis, T., & Rowstron, A. (2011) · Proceedings of the ACM SIGCOMM 2011 conference · doi:10.1145/2018436.2018465 · Cited by 750
Map-reduce-merge: simplified relational data processing on large clusters
Yang, H.-C., Dasdan, A., Hsiao, R.-L., & Parker, D. S. (2007) · Proceedings of the 2007 ACM SIGMOD international conference on Management of data · doi:10.1145/1247480.1247602 · Cited by 715
HaLoop: efficient iterative data processing on large clusters
Bu, Y., Howe, B., Balazinska, M., & Ernst, M. D. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1920881 · Cited by 693
Naiad: a timely dataflow system
Murray, D. G., McSherry, F., Isaacs, R., Isard, M., Barham, P., & Abadi, M. (2013) · Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles · doi:10.1145/2517349.2522738 · Cited by 656
Bahmani, B., Moseley, B., Vattani, A., Kumar, R., & Vassilvitskii, S. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2180912.2180915 · Cited by 628
FAWN: a fast array of wimpy nodes
Andersen, D. G., Franklin, J., Kaminsky, M., Phanishayee, A., Tan, L., & Vasudevan, V. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629577 · Cited by 561
Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis
Fu, Q., Lou, J.-G., Wang, Y., & Li, J. (2009) · 2009 Ninth IEEE International Conference on Data Mining · doi:10.1109/icdm.2009.60 · Cited by 542
Dong, X. L., & Srivastava, D. (2015) · Synthesis Lectures on Data Management · doi:10.1007/978-3-031-01853-4 · Cited by 539
RDMA over Commodity Ethernet at Scale
Guo, C., Wu, H., Deng, Z., Soni, G., Ye, J., Padhye, J., & Lipshteyn, M. (2016) · Proceedings of the 2016 ACM SIGCOMM Conference · doi:10.1145/2934872.2934908 · Cited by 497
Searching for SNPs with cloud computing
Langmead, B., Schatz, M. C., Lin, J., Pop, M., & Salzberg, S. L. (2009) · Genome Biology · doi:10.1186/gb-2009-10-11-r134 · Cited by 484
Recent Advances in Graph Partitioning
Buluç, A., Meyerhenke, H., Safro, I., Sanders, P., & Schulz, C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49487-6_4 · Cited by 482
Scientific data management in the coming decade
Gray, J., Liu, D. T., Nieto-Santisteban, M., Szalay, A., DeWitt, D. J., & Heber, G. (2005) · ACM SIGMOD Record · doi:10.1145/1107499.1107503 · Cited by 482
Interactive dynamics for visual analysis
Heer, J., & Shneiderman, B. (2012) · Communications of the ACM · doi:10.1145/2133806.2133821 · Cited by 474
A Fundamental Tradeoff Between Computation and Communication in Distributed Computing
Li, S., Maddah-Ali, M. A., Yu, Q., & Avestimehr, A. S. (2018) · IEEE Transactions on Information Theory · doi:10.1109/tit.2017.2756959 · Cited by 451
Efficient coflow scheduling with Varys
Chowdhury, M., Zhong, Y., & Stoica, I. (2014) · Proceedings of the 2014 ACM conference on SIGCOMM · doi:10.1145/2619239.2626315 · Cited by 438
A survey of open source tools for machine learning with big data in the Hadoop ecosystem
Landset, S., Khoshgoftaar, T. M., Richter, A. N., & Hasanin, T. (2015) · Journal of Big Data · doi:10.1186/s40537-015-0032-1 · Cited by 432
An overview of the Hadoop/MapReduce/HBase framework and its current applications in bioinformatics
Taylor, R. C. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-s12-s1 · Cited by 431
MapReduce and parallel DBMSs: friends or foes?
Stonebraker, M., Abadi, D., DeWitt, D. J., Madden, S., Paulson, E., Pavlo, A., & Rasin, A. (2010) · Communications of the ACM · doi:10.1145/1629175.1629197 · Cited by 430
Data warehousing and analytics infrastructure at facebook
Thusoo, A., Shao, Z., Anthony, S., Borthakur, D., Jain, N., Sen Sarma, J., Murthy, R., & Liu, H. (2010) · Proceedings of the 2010 ACM SIGMOD International Conference on Management of data · doi:10.1145/1807167.1807278 · Cited by 425
Qian, L., Luo, Z., Du, Y., & Guo, L. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-10665-1_63 · Cited by 415
Building a high-level dataflow system on top of Map-Reduce: the Pig experience
Gates, A. F., Natkovich, O., Chopra, S., Kamath, P., Narayanamurthy, S. M., Olston, C., Reed, B., Srinivasan, S., & Srivastava, U. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687553.1687568 · Cited by 405
Large-scale incremental processing using distributed transactions and notifications
Peng, D. N., & Dabek, F. (2010) · doi:10.5555/1924943.1924961 · Cited by 402
Mahout: Low-overhead datacenter traffic management using end-host-based elephant detection
Curtis, A. R., Kim, W., & Yalagandula, P. (2011) · 2011 Proceedings IEEE INFOCOM · doi:10.1109/infcom.2011.5934956 · Cited by 402
Big data privacy: a technological perspective and review
Jain, P., Gyanchandani, M., & Khare, N. (2016) · Journal of Big Data · doi:10.1186/s40537-016-0059-y · Cited by 392
Pingmesh: A Large-Scale System for Data Center Network Latency Measurement and Analysis
Guo, C., Yuan, L., Xiang, D., Dang, Y., Huang, R., Maltz, D., Liu, Z., Wang, V., Pang, B., Chen, H., et al. (2015) · Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication · doi:10.1145/2785956.2787496 · Cited by 390
When private set intersection meets big data: an efficient and scalable protocol
Dong, C., Chen, L., & Wen, Z. (2013) · Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security - CCS '13 · doi:10.1145/2508859.2516701 · Cited by 387
Big Data and supply chain management: A review and bibliometric analysis
Mishra, D., Gunasekaran, A., Papadopoulos, T., & Childe, S. J. (2018) · Annals of Operations Research · doi:10.1007/s10479-016-2236-y · Cited by 382
A review of automatic selection methods for machine learning algorithms and hyper-parameter values
Luo, G. (2016) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-016-0125-6 · Cited by 357
Coflow: a networking abstraction for cluster applications
Chowdhury, M., & Stoica, I. (2012) · Proceedings of the 11th ACM Workshop on Hot Topics in Networks · doi:10.1145/2390231.2390237 · Cited by 340
Tirmazi, M., Barker, A., Deng, N., Haque, M. E., Qin, Z. G., Hand, S., Harchol-Balter, M., & Wilkes, J. (2020) · Proceedings of the Fifteenth European Conference on Computer Systems · doi:10.1145/3342195.3387517 · Cited by 325
Autopilot: workload autoscaling at Google
Rzadca, K., Findeisen, P., Swiderski, J., Zych, P., Broniek, P., Kusmierek, J., Nowak, P., Strack, B., Witusowski, P., Hand, S., et al. (2020) · Proceedings of the Fifteenth European Conference on Computer Systems · doi:10.1145/3342195.3387524 · Cited by 303
Seven V's of Big Data understanding Big Data to extract value
Khan, M. A.-U.-D., Uddin, M. F., & Gupta, N. (2014) · Proceedings of the 2014 Zone 1 Conference of the American Society for Engineering Education · doi:10.1109/aseezone1.2014.6820689 · Cited by 290
In-Network Computation is a Dumb Idea Whose Time Has Come
Sapio, A., Abdelaziz, I., Aldilaijan, A., Canini, M., & Kalnis, P. (2017) · Proceedings of the 16th ACM Workshop on Hot Topics in Networks · doi:10.1145/3152434.3152461 · Cited by 278
Heer, J., & Shneiderman, B. (2012) · Queue · doi:10.1145/2133416.2146416 · Cited by 269
Practical Near-Data Processing for In-Memory Analytics Frameworks
Gao, M., Ayers, G., & Kozyrakis, C. (2015) · 2015 International Conference on Parallel Architecture and Compilation (PACT) · doi:10.1109/pact.2015.22 · Cited by 266
Fast Counting of Triangles in Large Real Networks without Counting: Algorithms and Laws
Tsourakakis, C. E. (2008) · 2008 Eighth IEEE International Conference on Data Mining · doi:10.1109/icdm.2008.72 · Cited by 265
Nephele/PACTs: a programming model and execution framework for web-scale analytical processing
Battré, D., Ewen, S., Hueske, F., Kao, O., Markl, V., & Warneke, D. (2010) · Proceedings of the 1st ACM symposium on Cloud computing · doi:10.1145/1807128.1807148 · Cited by 261
Life beyond big data–governing with little analytics
Amoore, L., & Piotukh, V. (2015) · Economy and Society · doi:10.1080/03085147.2015.1043793 · Cited by 187
A distributed spatial method for modeling maritime routes
Zissis, D., Chatzikokolakis, K., Spiliopoulos, G., & Vodas, M. (2020) · IEEE Access · doi:10.1109/access.2020.2979612 · Cited by 53
“Due to the volume of related data, a distributed MapReduce programming approach is adopted for this step, executed on a spark cluster [37].”
MethodICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems
Simard, P., Chickering, D., Lakshmiratan, A., Charles, D., Bottou, L., Suarez, C. G. J., Grangier, D., Amershi, S., Verwey, J., & Suh, J. (2014) · arXiv · doi:10.48550/arxiv.1409.4814 · Cited by 15
Metadata
- Authors
- Senior Software Developer, A. U., & Dayalan, M.
- Venue
- International Journal of Research and Engineering
- Publisher
- Marwah Infotech
- Published
- 2018-04
- Volume
- 5
- Issue
- 5
- Pages
- 399-403
- ISSN
- 2348-7852 (ISSN-L)
- 2348-7860
- OpenAlex
- W2122465391
- Cited by
- 2,948
- References
- 13
- FWCI
- 215.33
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.