Extreme Binning: Scalable, parallel deduplication for chunk-based file backup
The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Data deduplication is an essential and critical component of backup systems. Essential, because it reduces storage space requirements, and critical, because the performance of the entire backup operation depends on its throughput. Traditional backup workloads consist of large data streams with high locality, which existing deduplication techniques require to provide reasonable throughput. We present Extreme Binning, a scalable deduplication technique for non-traditional backup workloads that are made up of individual files with no locality among consecutive files in a given window of time. Due to lack of locality, existing techniques perform poorly on these workloads. Extreme Binning exploits file similarity instead of locality, and makes only one disk access for chunk lookup per file, which gives reasonable throughput. Multi-node backup systems built with Extreme Binning scale gracefully with the amount of input data; more backup nodes can be added to boost throughput. Each file is allocated using a stateless routing algorithm to only one node, allowing for maximum parallelization, and each backup node is autonomous with no dependency across nodes, making data management tasks robust with low overhead.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Chord: A scalable peer-to-peer lookup service for internet applications
Stoica, I., Morris, R., Karger, D., Kaashoek, M. F., & Balakrishnan, H. (2001) · Proceedings of the 2001 conference on Applications, technologies, architectures, and protocols for computer communications · doi:10.1145/383059.383071 · Cited by 9,623
Space/time trade-offs in hash coding with allowable errors
Bloom, B. H. (1970) · Communications of the ACM · doi:10.1145/362686.362692 · Cited by 7,492
Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems
Rowstron, A., & Druschel, P. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-45518-3_18 · Cited by 7,276
The MD5 Message-Digest Algorithm
Rivest (1992) · Medical Entomology and Zoology · Cited by 2,853
Étude comparative de la distribution florale dans une portion des Alpes et du Jura
Jaccard, P. (1901) · E-Periodica · doi:10.5169/seals-266450 · Cited by 2,353
OceanStore: an architecture for global-scale persistent storage
Kubiatowicz, J., Bindel, D., Chen, Y., Czerwinski, S., Eaton, P., Geels, D., Gummadi, R., Rhea, S., Weatherspoon, H., Weimer, W., et al. (2000) · ACM SIGPLAN Notices · doi:10.1145/356989.357007 · Cited by 2,002
The MD5 Message-Digest Algorithm
Rivest, R. (1992) · doi:10.17487/rfc1321 · Cited by 1,976
Tapestry: A Resilient Global-Scale Overlay for Service Deployment
Zhao, B. Y., Huang, L., Stribling, J., Rhea, S. C., Joseph, A. D., & Kubiatowicz, J. D. (2004) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2003.818784 · Cited by 1,730
On the resemblance and containment of documents
Broder, A. Z. (2002) · Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171) · doi:10.1109/sequen.1997.666900 · Cited by 1,698
OceanStore: an architecture for global-scale persistent storage
Kubiatowicz, J., Bindel, D., Chen, Y., Czerwinski, S., Eaton, P., Geels, D., Gummadi, R., Rhea, S., Weatherspoon, H., Weimer, W., et al. (2000) · Proceedings of the ninth international conference on Architectural support for programming languages and operating systems · doi:10.1145/378993.379239 · Cited by 869
A low-bandwidth network file system
Muthitacharoen, A., Chen, B., & Mazières, D. (2001) · Proceedings of the eighteenth ACM symposium on Operating systems principles · doi:10.1145/502034.502052 · Cited by 784
Venti: A New Approach to Archival Storage
Quinlan, S., & Dorward, S. (2002) · Cited by 732
Cox, L. P., Murray, C. D., & Noble, B. D. (2002) · ACM SIGOPS Operating Systems Review · doi:10.1145/844128.844155 · Cited by 334
LH*—a scalable, distributed data structure
Litwin, W., Neimat, M.-A., & Schneider, D. A. (1996) · ACM Transactions on Database Systems · doi:10.1145/236711.236713 · Cited by 236
Shark: scaling file servers via cooperative caching
Annapureddy, S., Freedman, M. J., & Mazières, D. (2005) · Networked Systems Design and Implementation · doi:10.5555/1251203.1251213 · Cited by 154
Application-specific Delta-encoding via Resemblance Detection.
Douglis, F., & Iyengar, A. (2003) · Cited by 134
Honicky, R. J., & Miller, E. L. (2004) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2004.1303042 · Cited by 128
The design of a similarity based deduplication system
Aronovich, L., Asher, R., Bachmat, E., Bitner, H., Hirsch, M., & Klein, S. T. (2009) · doi:10.1145/1534530.1534539 · Cited by 120
Fast, inexpensive content-addressed storage in foundation
Rhea, S. C., Cox, R., & Pesterev, A. A. (2008) · Cited by 100
Finding similar files in large document repositories
Forman, G., Eshghi, K., & Chiocchetti, S. (2005) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1081870.1081916 · Cited by 91
Self-citationAn analysis of compare-by-hash
Henson, V. (2003) · Workshop on Hot Topics in Operating Systems · Cited by 67
Compare-by-hash: a reasoned analysis
Black, J. (2006) · USENIX Annual Technical Conference · Cited by 53
Bhagwat, D., Eshghi, K., & Mehra, P. (2007) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1281192.1281207 · Cited by 25
Self-citation
Cited by
A study of practical deduplication
Meyer, D. T., & Bolosky, W. J. (2012) · ACM Transactions on Storage · doi:10.1145/2078861.2078864 · Cited by 476
A Comprehensive Study of the Past, Present, and Future of Data Deduplication
Xia, W., Jiang, H., Feng, D., Douglis, F., Shilane, P., Hua, Y., Fu, M., Zhang, Y., & Zhou, Y. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2016.2571298 · Cited by 318
Investigating the Adoption of Hybrid Encrypted Cloud Data Deduplication With Game Theory
Liang, X., Yan, Z., Deng, R. H., & Zheng, Q. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.3028685 · Cited by 18
Metadata
- Authors
- Bhagwat, D., Eshghi, K., Long, D. D. E., & Lillibridge, M.
- Venue
- 2009 IEEE International Symposium on Modeling, Analysis & Simulation of Computer and Telecommunication Systems
- Publisher
- IEEE
- Published
- 2009-09
- Pages
- 1-9
- OpenAlex
- W1969126835
- Cited by
- 325
- References
- 23
- FWCI
- 16.19
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.