The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
The Hadoop Distributed File System
Abstract
The Hadoop Distributed File System (HDFS) is designed to store very large data sets reliably, and to stream those data sets at high bandwidth to user applications. In a large cluster, thousands of servers both host directly attached storage and execute user application tasks. By distributing storage and computation across many servers, the resource can grow with demand while remaining economical at every size. We describe the architecture of HDFS and report on experience using HDFS to manage 25 petabytes of enterprise data at Yahoo!.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003) · ACM SIGOPS Operating Systems Review · doi:10.1145/1165389.945450 · Cited by 4,984
MapReduce: Simplified Data Processing on Large Cluster
Senior Software Developer, A. U., & Dayalan, M. (2018) · International Journal of Research and Engineering · doi:10.21276/ijre.2018.5.5.4 · Cited by 2,948
White, T. (2009) · Cited by 2,800
Hive: a warehousing solution over a map-reduce framework
Thusoo, A., Sarma, J. S., Jain, N., Shao, Z., Chakka, P., Anthony, S., Liu, H., Wyckoff, P., & Murthy, R. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687553.1687609 · Cited by 1,556
Ceph: a scalable, high-performance distributed file system
Weil, S. A., Brandt, S., Miller, E. L., Long, D. D. E., & Maltzahn, C. (2006) · eScholarship (California Digital Library) · doi:10.5555/1298455.1298485 · Cited by 1,402
Ghemawat, S., Gobioff, H., & Leung, S.-T. A. (2003) · doi:10.1145/945449.945450 · Cited by 1,347
PVFS : a parallel file system for linux clusters
Carns, P., Ligon, W. B., III, Ross, R. B., & Thakur, R. (2000) · University of North Texas Digital Library (University of North Texas) · Cited by 850
Building a high-level dataflow system on top of Map-Reduce: the Pig experience
Gates, A. F., Natkovich, O., Chopra, S., Kamath, P., Narayanamurthy, S. M., Olston, C., Reed, B., Srinivasan, S., & Srivastava, U. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687553.1687568 · Cited by 405
McKusick, K., & Quinlan, S. (2010) · Communications of the ACM · doi:10.1145/1666420.1666439 · Cited by 124
GFS: Evolution on Fast-forward
McKusick, M. K., & Quinlan, S. (2009) · Queue · doi:10.1145/1594204.1594206 · Cited by 72
Research works printed but not linked: 6
- 10.1145/155848.155861 · doi:10.1145/155848.155861
- 10.1109/SDNE.1994.337764 · doi:10.1109/sdne.1994.337764
- tantisiriroj, 2008, Data-intensive file systems for Internet services: A rose by any print name …, Technical Report CMUPDL-08-114 Parallel Data Laboratory
- 10.1007/978-1-4302-1943-9 · doi:10.1007/978-1-4302-1943-9
- welch, 2008, Scalable Performance of the Panasas Parallel file System, Proc 4th USENIX Conference on File and Storage Technologies
- 0, Lustre file system
Cited by
Edge computing: Vision and challenges
Shi, W., Cao, J., Zhang, Q., Li, Y., & Xu, L. (2016) · IEEE Internet of Things Journal · doi:10.1109/jiot.2016.2579198 · Cited by 8,733
The rise of big data on cloud computing: Review and open research issues
Hashem, I. A. T., Yaqoob, I., Anuar, N. B., Mokhtar, S., Gani, A., & Ullah Khan, S. (2015) · Information Systems · doi:10.1016/j.is.2014.07.006 · Cited by 2,656
Apache Hadoop YARN: yet another resource negotiator
Vavilapalli, V. K., Murthy, A. C., Douglas, C., Agarwal, S., Konar, M., Evans, R., Graves, T., Lowe, J., Shah, H., Seth, S., et al. (2013) · Proceedings of the 4th annual Symposium on Cloud Computing · doi:10.1145/2523616.2523633 · Cited by 1,848
Self-citationBig data in healthcare: management, analysis and future prospects
Dash, S., Shakyawar, S. K., Sharma, M., & Kaushik, S. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0217-0 · Cited by 1,747
A review of clustering techniques and developments
Saxena, A., Prasad, M., Gupta, A., Bharill, N., Patel, O. P., Tiwari, A., Er, M. J., Ding, W., & Lin, C.-T. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.06.053 · Cited by 1,357
Machine learning with big data: challenges and approaches.
L'Heureux, A., Grolinger, K., Elyamany, H. F., & Capretz, M. A. M. (2017) · IEEE Access · doi:10.1109/access.2017.2696365 · Cited by 1,023
A formal definition of Big Data based on its essential features
De Mauro, A., Greco, M., & Grimaldi, M. (2016) · Library Review · doi:10.1108/lr-06-2015-0061 · Cited by 930
A Survey on Distributed Machine Learning
Verbraeken, J., Wolting, M., Katzy, J., Kloppenburg, J., Verbelen, T., & Rellermeyer, J. S. (2021) · ACM Computing Surveys · doi:10.1145/3377454 · Cited by 872
Big Data and cloud computing: innovation opportunities and challenges
Yang, C., Huang, Q., Li, Z., Liu, K., & Hu, F. (2017) · International Journal of Digital Earth · doi:10.1080/17538947.2016.1239771 · Cited by 838
The use of machine learning algorithms in recommender systems: a systematic review
Portugal, I., Alencar, P., & Cowan, D. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.12.020 · Cited by 743
“The proposed architecture that provides on demand tools for analysis uses the storage technologies HDFS (Hadoop Distributed File System) (Shvachko et al., 2010) and HBase6 for persistence logs and structured information about the execution and predictions.”
MethodMachine Learning Operations (MLOps): Overview, Definition, and Architecture
Kreuzberger, D., Kühl, N., & Hirschl, S. (2023) · IEEE Access · doi:10.1109/access.2023.3262138 · Cited by 766
“file systems [41], [42], or distributed databases [43], [44] combined with parallel and distributed data processing algorithms (e.”
Díaz, M., Martín, C., & Rubio, B. (2016) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2016.01.010 · Cited by 689
Big data for development: a review of promises and challenges
Hilbert, M. (2016) · Development Policy Review · doi:10.1111/dpr.12142 · Cited by 679
Parallel K-Means Clustering Based on MapReduce
Zhao, W., Ma, H., & He, Q. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-10665-1_71 · Cited by 626
What is big data? A consensual definition and a review of key research topics
De Mauro, A., Greco, M., & Grimaldi, M. (2015) · AIP Conference Proceedings · doi:10.1063/1.4907823 · Cited by 529
Big data for cyber physical systems in industry 4.0: a survey
Xu, L. D., & Duan, L. (2019) · Enterprise Information Systems · doi:10.1080/17517575.2018.1442934 · Cited by 526
(2010) · Springer eBooks · doi:10.1007/978-1-4419-6524-0 · Cited by 519
Aceto, G., Persico, V., & Pescapé, A. (2018) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2018.02.008 · Cited by 504
Big Data Analytics in Healthcare
Belle, A., Thiagarajan, R., Soroushmehr, S. M. R., Navidi, F., Beard, D. A., & Najarian, K. (2015) · BioMed Research International · doi:10.1155/2015/370194 · Cited by 487
Industrial Internet: A Survey on the Enabling Technologies, Applications, and Challenges
Li, J.-Q., Yu, F. R., Deng, G., Luo, C., Ming, Z., & Yan, Q. (2017) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2017.2691349 · Cited by 472
Context-Aware Computing, Learning, and Big Data in Internet of Things: A Survey
Sezer, O. B., Dogdu, E., & Ozbayoglu, A. M. (2018) · IEEE Internet of Things Journal · doi:10.1109/jiot.2017.2773600 · Cited by 453
Trinity: a distributed graph engine on a memory cloud
Shao, B., Wang, H., & Li, Y. (2013) · Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2463676.2467799 · Cited by 432
In-Memory Big Data Management and Processing: A Survey
Zhang, H., Chen, G., Ooi, B. C., Tan, K.-L., & Zhang, M. (2015) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2015.2427795 · Cited by 414
Occupy the cloud: distributed computing for the 99%
Jonas, E., Pu, Q., Venkataraman, S., Stoica, I., & Recht, B. (2017) · Proceedings of the 2017 Symposium on Cloud Computing · doi:10.1145/3127479.3128601 · Cited by 411
An IoT-Oriented Data Storage Framework in Cloud Computing Platform
Jiang, L., Xu, L. D., Cai, H., Jiang, Z., Bu, F., & Xu, B. (2014) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2014.2306384 · Cited by 352
Cloud-enabled prognosis for manufacturing
Gao, R., Wang, L., Teti, R., Dornfeld, D., Kumara, S., Mori, M., & Helu, M. (2015) · CIRP Annals · doi:10.1016/j.cirp.2015.05.011 · Cited by 351
Blockchain for Large-Scale Internet of Things Data Storage and Protection
Li, R., Song, T., Mei, B., Li, H., Cheng, X., & Sun, L. (2019) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2018.2853167 · Cited by 343
Manufacturing big data ecosystem: A systematic literature review
Cui, Y., Kara, S., & Chan, K. C. (2020) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2019.101861 · Cited by 334
Adding Virtualization Capabilities to the Grid’5000 Testbed
Balouek, D., Amarie, A. C., Charrier, G., Desprez, F., Jeannot, E., Jeanvoine, E., Lèbre, A., Margery, D., Niclausse, N., Nussbaum, L., et al. (2013) · Communications in Computer and Information Science · doi:10.1007/978-3-319-04519-1_1 · Cited by 315
“Short-Dot”: Computing Large Linear Transforms Distributedly Using Coded Short Dot Products
Dutta, S., Cadambe, V., & Grover, P. (2019) · IEEE Transactions on Information Theory · doi:10.1109/tit.2019.2927558 · Cited by 313
A big data-driven framework for sustainable and smart additive manufacturing
Majeed, A., Zhang, Y., Ren, S., Lv, J., Peng, T., Waqar, S., & Yin, E. (2021) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2020.102026 · Cited by 313
An Overview of Platforms for Big Earth Observation Data Management and Analysis
Gomes, V., Queiroz, G., & Ferreira, K. (2020) · Remote Sensing · doi:10.3390/rs12081253 · Cited by 307
Mapping brain activity at scale with cluster computing
Freeman, J., Vladimirov, N., Kawashima, T., Mu, Y., Sofroniew, N. J., Bennett, D. V., Rosen, J., Yang, C.-T., Looger, L. L., & Ahrens, M. B. (2014) · Nature Methods · doi:10.1038/nmeth.3041 · Cited by 299
Understandable Big Data: A survey
Kacfah Emani, C., Cullot, N., & Nicolle, C. (2015) · Computer Science Review · doi:10.1016/j.cosrev.2015.05.002 · Cited by 299
Splitwise: Efficient Generative LLM Inference Using Phase Splitting
Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R. (2024) · 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca59077.2024.00019 · Cited by 298
“Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
BackgroundA Survey of Big Data Analytics in Healthcare and Government
Archenaa, J., & Anita, E. A. M. (2015) · Procedia Computer Science · doi:10.1016/j.procs.2015.04.021 · Cited by 296
Materials Data Science: Current Status and Future Outlook
Kalidindi, S. R., & De Graef, M. (2015) · Annual Review of Materials Research · doi:10.1146/annurev-matsci-070214-020844 · Cited by 294
Bhattarai, B. P., Paudyal, S., Luo, Y., Mohanpurkar, M., Cheung, K., Tonkoski, R., Hovsapian, R., Myers, K. S., Zhang, R., Zhao, P., et al. (2019) · IET Smart Grid · doi:10.1049/iet-stg.2018.0261 · Cited by 292
(2019) · NORA - Norwegian Open Research Archives · Cited by 291
A survey of data partitioning and sampling methods to support big data analysis
Mahmud, M. S., Huang, J. Z., Salloum, S., Emara, T. Z., & Sadatdiynov, K. (2020) · Big Data Mining and Analytics · doi:10.26599/bdma.2019.9020015 · Cited by 282
González-Granadillo, G., González-Zarzosa, S., & Diaz, R. (2021) · Sensors · doi:10.3390/s21144759 · Cited by 280
Cloud-enabled wireless body area networks for pervasive healthcare
Wan, J., Zou, C., Ullah, S., Lai, C.-F., Zhou, M., & Wang, X. (2013) · IEEE Network · doi:10.1109/mnet.2013.6616116 · Cited by 271
Leveraging big data analytics in healthcare enhancement: trends, challenges and opportunities
Rehman, A., Naz, S., & Razzak, I. (2022) · Multimedia Systems · doi:10.1007/s00530-020-00736-8 · Cited by 264
Rapyuta: A Cloud Robotics Platform
Mohanarajah, G., Hunziker, D., D'Andrea, R., & Waibel, M. (2015) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2014.2329556 · Cited by 255
Chen, J., Yi, C., Okegbile, S. D., Cai, J., & Shen, X. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3308717 · Cited by 254
New Horizons for a Data-Driven Economy
(2016) · Springer eBooks · doi:10.1007/978-3-319-21569-3 · Cited by 253
(2015) · doi:10.1145/2815400.2815415 · Cited by 202
Best practices for analyzing large-scale health data from wearables and smartphone apps
Hicks, J. L., Althoff, T., Sosic, R., Kuhar, P., Bostjancic, B., King, A. C., Leskovec, J., & Delp, S. L. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0121-1 · Cited by 183
Ahmad, S., Shakeel, I., Mehfuz, S., & Ahmad, J. (2023) · Computer Science Review · doi:10.1016/j.cosrev.2023.100568 · Cited by 197
Ramírez-Gallego, S., Fernández, A., García, S., Chen, M., & Herrera, F. (2018) · Information Fusion · doi:10.1016/j.inffus.2017.10.001 · Cited by 140
“In HDFS, the fi rst block is ensured to be placed in the same processing node, whereas the other two replicas are sent to di ff erent racks to prevent abrupt ends due to inter-rack issues.”
MethodInfluentialGraph-based root cause analysis for service-oriented and microservice architectures
Brandón, Á., Solé, M., Huélamo, A., Solans, D., Pérez, M. S., & Muntés-Mulero, V. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2019.110432 · Cited by 121
Sifter: Scalable Sampling for Distributed Traces, without Feature Engineering
Las-Casas, P., Papakerashvili, G., Anand, V., & Mace, J. (2019) · Proceedings of the ACM Symposium on Cloud Computing · doi:10.1145/3357223.3362736 · Cited by 42
Distributed temporal graph analytics with GRADOOP
Rost, C., Gomez, K., Täschner, M., Fritzsche, P., Schons, L., Christ, L., Adameit, T., Junghanns, M., & Rahm, E. (2022) · The VLDB Journal · doi:10.1007/s00778-021-00667-4 · Cited by 39
Industrial Big Data Analytics: Challenges and Opportunities
Al-Abassi, A., Karimipour, H., HaddadPajouh, H., Dehghantanha, A., & Parizi, R. M. (2020) · Handbook of Big Data Privacy · doi:10.1007/978-3-030-38557-6_3 · Cited by 32
Kazman, R., Goldenson, D., Monarch, I., Nichols, W., & Valetto, G. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2465387 · Cited by 31
Towards continuous policy-driven demand response in data centers
Irwin, D., Sharma, N., & Shenoy, P. (2011) · Proceedings of the 2nd ACM SIGCOMM workshop on Green networking · doi:10.1145/2018536.2018541 · Cited by 22
Computational Strategies for Scalable Genomics Analysis
Shi, L., & Wang, Z. (2019) · Genes · doi:10.3390/genes10121017 · Cited by 19
LLM meets ML: Data-efficient Anomaly Detection on Unstable Logs
Hadadi, F., Xu, Q., Bianculli, D., & Briand, L. (2024) · arXiv · doi:10.48550/arxiv.2406.07467 · Cited by 7
Large-Scale Neo-Heterogeneous Programming and Optimization of SNP Detection on Tianhe-2
Cui, Y., Liao, X., Peng, S., Lu, Y., Yang, C., Wang, B., & Wu, C. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-20119-1_6 · Cited by 4
“Crossbow [19] is a parallel solution using Hadoop [20] and accelerates detection with cloud computing.”
Method
Metadata
- Authors
- Shvachko, K., Kuang, H., Radia, S., & Chansler, R.
- Venue
- 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST)
- Publisher
- IEEE
- Published
- 2010-05
- Pages
- 1-10
- OpenAlex
- W2119738171
- Cited by
- 4,834
- References
- 12
- FWCI
- 124.13
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.