The DOI matches its Crossref record.
DisCo: Distributed Co-clustering with Map-Reduce: A Case Study towards Petabyte-Scale End-to-End Mining
Abstract
Huge datasets are becoming prevalent; even as researchers, we now routinely have to work with datasets that are up to a few terabytes in size. Interesting real-world applications produce huge volumes of messy data. The mining process involves several steps, starting from pre-processing the raw data to estimating the final models. As data become more abundant, scalable and easy-to-use tools for distributed processing are also emerging. Among those, Map-Reduce has been widely embraced by both academia and industry. In database terms, Map-Reduce is a simple yet powerful execution engine, which can be complemented with other data storage and management components, as necessary. In this paper we describe our experiences and findings in applying Map-Reduce, from raw data to final models, on an important mining task. In particular, we focus on co-clustering, which has been studied in many applications such as text mining, collaborative filtering, bio-informatics, graph mining. We propose the distributed co-clustering (DisCo) framework, which introduces practical approaches for distributed data pre-processing, and co-clustering. We develop DisCo using Hadoop, an open source Map-Reduce implementation. We show that DisCo can scale well and efficiently process and analyze extremely large datasets (up to several hundreds of gigabytes) on commodity hardware.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Wu, X., Zhu, X., Wu, G.-Q., & Ding, W. (2014) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2013.109 · Cited by 2,410
Graph based anomaly detection and description: a survey
Akoglu, L., Tong, H., & Koutra, D. (2015) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-014-0365-y · Cited by 1,504
“, 2003], and its distributed variants [Papadimitriou and Sun, 2008]; dynamic community detection algorithms Tantipathananandh and Berger-Wolf [2009, 2011], Tantipathananandh et al.”
BackgroundMethodSocial influence analysis in large-scale networks
Tang, J., Sun, J., Wang, C., & Yang, Z. (2009) · Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1557019.1557108 · Cited by 927
Self-citationPEGASUS: A Peta-Scale Graph Mining System Implementation and Observations
Kang, U., Tsourakakis, C. E., & Faloutsos, C. (2009) · 2009 Ninth IEEE International Conference on Data Mining · doi:10.1109/icdm.2009.14 · Cited by 628
CopyCatch: stopping group attacks by spotting lockstep behavior in social networks
Beutel, A., Xu, W., Guruswami, V., Palow, C., & Faloutsos, C. (2013) · Proceedings of the 22nd international conference on World Wide Web · doi:10.1145/2488388.2488400 · Cited by 335
Metadata
- Authors
- Papadimitriou, S., & Sun, J.
- Venue
- 2008 Eighth IEEE International Conference on Data Mining
- Publisher
- IEEE
- Published
- 2008-12
- Pages
- 512-521
- ISSN
- 1550-4786 (ISSN-L)
- OpenAlex
- W2127164716
- Cited by
- 209
- References
- 0
- FWCI
- 11.80
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: DisCo: Distributed Co-clustering with Map-Reduce: A Case Study towards Petabyte-Scale End-to-End Mining. https://app.cochat.ai/papers/10.1109/icdm.2008.142
- The paper
- Papadimitriou, S., & Sun, J. (2008). DisCo: Distributed Co-clustering with Map-Reduce: A Case Study towards Petabyte-Scale End-to-End Mining. 2008 Eighth IEEE International Conference on Data Mining. https://doi.org/10.1109/icdm.2008.142
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.