The DOI matches its Crossref record; 93 of the 93 references it cites were found in a registry or an index.
A Comprehensive Study of the Past, Present, and Future of Data Deduplication
Abstract
Data deduplication, an efficient approach to data reduction, has gained increasing attention and popularity in large-scale storage systems due to the explosive growth of digital data. It eliminates redundant data at the file or subfile level and identifies duplicate content by its cryptographically secure hash signature (i.e., collision-resistant fingerprint), which is shown to be much more computationally efficient than the traditional compression approaches in large-scale storage systems. In this paper, we first review the background and key features of data deduplication, then summarize and classify the state-of-the-art research in data deduplication according to the key workflow of the data deduplication process. The summary and taxonomy of the state of the art on deduplication help identify and understand the most important design considerations for data deduplication systems. In addition, we discuss the main applications and industry trend of data deduplication, and provide a list of the publicly available sources for deduplication research and studies. Finally, we outline the open problems and future research directions facing deduplication-based storage systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 48,949
Space/time trade-offs in hash coding with allowable errors
Bloom, B. H. (1970) · Communications of the ACM · doi:10.1145/362686.362692 · Cited by 7,492
A Method for the Construction of Minimum-Redundancy Codes
Huffman, D. (1952) · Proceedings of the IRE · doi:10.1109/jrproc.1952.273898 · Cited by 6,334
A universal algorithm for sequential data compression
Ziv, J., & Lempel, A. (1977) · IEEE Transactions on Information Theory · doi:10.1109/tit.1977.1055714 · Cited by 5,456
Compression of individual sequences via variable-rate coding
Ziv, J., & Lempel, A. (1978) · IEEE Transactions on Information Theory · doi:10.1109/tit.1978.1055934 · Cited by 3,432
JPEG2000: Image Compression Fundamentals, Standards and Practice
Taubman, D. S. (2002) · Journal of Electronic Imaging · doi:10.1117/1.1469618 · Cited by 3,159
Arithmetic coding for data compression
Witten, I. H., Neal, R. M., & Cleary, J. G. (1987) · Communications of the ACM · doi:10.1145/214762.214771 · Cited by 2,877
A Block-sorting Lossless Data Compression Algorithm
Burrows, M. T., & Wheeler, D. J. (1994) · Cited by 2,333
On the resemblance and containment of documents
Broder, A. Z. (2002) · Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171) · doi:10.1109/sequen.1997.666900 · Cited by 1,698
Merkle, R. C. (2007) · Lecture Notes in Computer Science · doi:10.1007/0-387-34805-0_21 · Cited by 1,691
A study of replacement algorithms for a virtual-storage computer
Belady, L. A. (1966) · IBM Systems Journal · doi:10.1147/sj.52.0078 · Cited by 1,659
Memory resource management in VMware ESX server
Waldspurger, C. A. (2002) · ACM SIGOPS Operating Systems Review · doi:10.1145/844128.844146 · Cited by 1,205
Pagh, R., & Rodler, F. F. (2004) · Journal of Algorithms · doi:10.1016/j.jalgor.2003.12.002 · Cited by 1,045
Rcs — a system for version control
Tichy, W. F. (1985) · Software: Practice and Experience · doi:10.1002/spe.4380150703 · Cited by 944
US Secure Hash Algorithm 1 (SHA1)
Eastlake, D., & Jones, P. (2001) · doi:10.17487/rfc3174 · Cited by 908
A low-bandwidth network file system
Muthitacharoen, A., Chen, B., & Mazières, D. (2001) · Proceedings of the eighteenth ACM symposium on Operating systems principles · doi:10.1145/502034.502052 · Cited by 784
Disk failures in the real world: what does an MTTF of 1,000,000 hours mean to you?
Schroeder, B., & Gibson, G. A. (2007) · Cited by 762
Christen, P. (2012) · Springer eBooks · doi:10.1007/978-3-642-31164-2 · Cited by 732
Venti: A New Approach to Archival Storage
Quinlan, S., & Dorward, S. (2002) · Cited by 732
DEFLATE Compressed Data Format Specification version 1.3
Deutsch, P. (1996) · doi:10.17487/rfc1951 · Cited by 675
Reclaiming space from duplicate files in a serverless distributed file system
Douceur, J. R., Adya, A., Bolosky, W. J., Simon, P., & Theimer, M. (2003) · Proceedings 22nd International Conference on Distributed Computing Systems · doi:10.1109/icdcs.2002.1022312 · Cited by 645
Message-Locked Encryption and Secure Deduplication
Bellare, M., Keelveedhi, S., & Ristenpart, T. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-38348-9_18 · Cited by 632
Finding similar files in a large file system
Manber, U. (1994) · Cited by 594
Proofs of ownership in remote storage systems
Halevi, S., Harnik, D., Pinkas, B., & Shulman-Peleg, A. (2011) · Proceedings of the 18th ACM conference on Computer and communications security · doi:10.1145/2046707.2046765 · Cited by 562
Secure Deduplication with Efficient and Reliable Convergent Key Management
Li, J., Chen, X., Li, M., Li, J., Lee, P. P. C., & Lou, W. (2014) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2013.284 · Cited by 549
An Introduction to Arithmetic Coding
Langdon, G. G. (1984) · IBM Journal of Research and Development · doi:10.1147/rd.282.0135 · Cited by 527
Side Channels in Cloud Services: Deduplication in Cloud Storage
Harnik, D., Pinkas, B., & Shulman-Peleg, A. (2010) · IEEE Security & Privacy Magazine · doi:10.1109/msp.2010.187 · Cited by 522
Feasibility of a serverless distributed file system deployed on an existing set of desktop PCs
Bolosky, W. J., Douceur, J. R., Ely, D., & Theimer, M. (2000) · Proceedings of the 2000 ACM SIGMETRICS international conference on Measurement and modeling of computer systems · doi:10.1145/339331.339345 · Cited by 484
A study of practical deduplication
Meyer, D. T., & Bolosky, W. J. (2012) · ACM Transactions on Storage · doi:10.1145/2078861.2078864 · Cited by 476
DupLESS: Server-Aided Encryption for Deduplicated Storage.
Bellare, M., Keelveedhi, S., & Ristenpart, T. (2013) · Cited by 456
Identifying and Filtering Near-Duplicate Documents
Broder, A. Z. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45123-4_1 · Cited by 395
Inside dropbox: understanding personal cloud storage services
Drago, I., Mellia, M., M. Munafo, M., Sperotto, A., Sadre, R., & Pras, A. (2012) · Proceedings of the 2012 Internet Measurement Conference · doi:10.1145/2398776.2398827 · Cited by 372
Cox, L. P., Murray, C. D., & Noble, B. D. (2002) · ACM SIGOPS Operating Systems Review · doi:10.1145/844128.844155 · Cited by 334
Extreme Binning: Scalable, parallel deduplication for chunk-based file backup
Bhagwat, D., Eshghi, K., Long, D. D. E., & Lillibridge, M. (2009) · 2009 IEEE International Symposium on Modeling, Analysis & Simulation of Computer and Telecommunication Systems · doi:10.1109/mascot.2009.5366623 · Cited by 325
Tridgell, A., & Mackerras, P. (1996) · ANU Open Research (Australian National University) · Cited by 310
A protocol-independent technique for eliminating redundant network traffic
Spring, N. T., & Wetherall, D. (2000) · ACM SIGCOMM Computer Communication Review · doi:10.1145/347057.347408 · Cited by 303
Storer, M. W., Greenan, K., Long, D. D. E., & Miller, E. L. (2008) · Proceedings of the 4th ACM international workshop on Storage security and survivability · doi:10.1145/1456469.1456471 · Cited by 288
Chen, F., Luo, T., & Zhang, X. (2011) · File and Storage Technologies · doi:10.5555/1960475.1960481 · Cited by 249
An Introduction to Duplicate Detection
Naumann, F., & Herschel, M. (2010) · Synthesis lectures on data management · doi:10.2200/s00262ed1v01y201003dtm003 · Cited by 245
FlashStore: high throughput persistent key-value store
Debnath, B. K., Sengupta, S., & Li, J. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1921015 · Cited by 237
Gupta, D., Lee, S., Vrable, M., Savage, S., Snoeren, A. C., Varghese, G., Voelker, G. M., & Vahdat, A. M. (2010) · Communications of the ACM · doi:10.1145/1831407.1831429 · Cited by 231
Secure Overlay Cloud Storage with Access Control and Assured Deletion
Tang, Y., Lee, P. P. C., Lui, J. C. S., & Perlman, R. (2012) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2012.49 · Cited by 231
Redundancy elimination within large collections of files
Kulkarni, P., Douglis, F., LaVoie, J., & Tracey, J. M. (2004) · Cited by 220
Self-citationSome applications of Rabin’s fingerprinting method
Broder, A. Z. (1993) · Sequences II · doi:10.1007/978-1-4613-9323-8_11 · Cited by 204
Dark clouds on the horizon: using cloud storage as attack vector and online slack space
Mulazzani, M., Schrittwieser, S., Leithner, M., Huber, M., & Weippl, E. R. (2011) · Cited by 194
Redundancy in network traffic: findings and implications
Anand, A., Muthukrishnan, C., Akella, A., & Ramjee, R. (2009) · Proceedings of the eleventh international joint conference on Measurement and modeling of computer systems · doi:10.1145/1555349.1555355 · Cited by 194
Xia, W., Jiang, H., Feng, D., & Hua, Y. (2011) · Cited by 187
Self-citationBoosting efficiency and security in proof of ownership for deduplication
Di Pietro, R., & Sorniotti, A. (2012) · Proceedings of the 7th ACM Symposium on Information, Computer and Communications Security · doi:10.1145/2414456.2414504 · Cited by 175
SmartRE: an architecture for coordinated network-wide redundancy elimination
Anand, A., Sekar, V., & Akella, A. (2009) · Proceedings of the ACM SIGCOMM 2009 conference on Data communication · doi:10.1145/1592568.1592580 · Cited by 175
I-CASH: Intelligently Coupled Array of SSD and HDD
Yang, Q., & Ren, J. (2011) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2011.5749736 · Cited by 161
Improving duplicate elimination in storage systems
Bobbarjung, D. R., Jagannathan, S., & Dubnicki, C. (2006) · ACM Transactions on Storage · doi:10.1145/1210596.1210599 · Cited by 145
Tradeoffs in scalable data routing for deduplication clusters
Dong, W., Douglis, F., Li, K., Patterson, H., Reddy, S., & Shilane, P. (2011) · doi:10.5555/1960475.1960477 · Cited by 138
Self-citationPrimary data deduplication-large scale study and system design
El-Shimi, A., Kalach, R., Kumar, A., et al. (2012) · Cited by 138
Demystifying data deduplication
Mandagere, N., Zhou, P., Smith, M. A., & Uttamchandani, S. (2008) · ACM/IFIP/USENIX international conference on Middleware · doi:10.1145/1462735.1462739 · Cited by 138
Bevan, A. (2015) · Antiquity · doi:10.15184/aqy.2015.102 · Cited by 135
Application-specific Delta-encoding via Resemblance Detection.
Douglis, F., & Iyengar, A. (2003) · Cited by 134
Self-citationA study on data deduplication in HPC storage systems
Meister, D., Kaiser, J., Brinkmann, A., Cortés, T., Kühn, M., & Kunkel, J. M. (2012) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc.2012.14 · Cited by 129
TAPER: tiered approach for eliminating redundancy in replica synchronization
Jain, N., Dahlin, M., & Tewari, R. (2005) · doi:10.5555/1251028.1251049 · Cited by 126
CDStore: Toward Reliable, Secure, and Cost-Efficient Cloud Storage via Convergent Dispersal
Li, M., Qin, C., Li, J., & Lee, P. P. C. (2016) · IEEE Internet Computing · doi:10.1109/mic.2016.45 · Cited by 121
Bimodal content defined chunking for backup streams
Kruus, E., Ungureanu, C., & Dubnicki, C. (2010) · File and Storage Technologies · doi:10.5555/1855511.1855529 · Cited by 120
WAN-optimized replication of backup datasets using stream-informed delta compression
Shilane, P., Huang, M., Wallace, G., & Hsu, W. (2012) · ACM Transactions on Storage · doi:10.1145/2385603.2385606 · Cited by 117
Self-citationVMFlock: virtual machine co-migration for the cloud
Al-Kiswany, S., Subhraveti, D., Sarkar, P., & Ripeanu, M. (2011) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/1996130.1996153 · Cited by 116
Exploiting Data Deduplication to Accelerate Live Virtual Machine Migration
Zhang, X., Huo, Z., Ma, J., et al. (2010) · Proceedings · doi:10.1109/cluster.2010.17 · Cited by 110
The design of a similarity based deduplication system
Aronovich, L., Asher, R., Bachmat, E., Bitner, H., Hirsch, M., & Klein, S. T. (2009) · doi:10.1145/1534530.1534539 · Cited by 109
Ddelta: A deduplication-inspired fast delta compression approach
Xia, W., Jiang, H., Feng, D., Tian, L., Fu, M., & Zhou, Y. (2014) · Performance Evaluation · doi:10.1016/j.peva.2014.07.016 · Cited by 101
Self-citationFast, inexpensive content-addressed storage in foundation
Rhea, S. C., Cox, R., & Pesterev, A. A. (2008) · Cited by 100
StoreGPU: exploiting graphics processing units to accelerate distributed storage systems
Al-Kiswany, S., Gharaibeh, A., Santos‐Neto, E., Yuan, G. L., & Ripeanu, M. (2008) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/1383422.1383443 · Cited by 95
Zhang, Y., Jiang, H., Feng, D., et al. (2015) · doi:10.1109/infocom.2015.7218510 · Cited by 92
Self-citationZhou, Y., Feng, D., Xia, W., Fu, M., Huang, F., Zhang, Y., & Li, C. (2015) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2015.7208297 · Cited by 89
Self-citationFeasibility of a serverless distributed file system deployed on an existing set of desktop PCs
Bolosky, W. J., Douceur, J. R., Ely, D., & Theimer, M. M. (2000) · ACM SIGMETRICS Performance Evaluation Review · doi:10.1145/345063.339345 · Cited by 83
dedupv1: Improving deduplication throughput using solid state drives (SSD)
Meister, D., & Brinkmann, A. (2010) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2010.5496992 · Cited by 77
WAN optimized replication of backup datasets using stream-informed delta compression
Shilane, P., Huang, M., Wallace, G., & Hsu, W. (2012) · File and Storage Technologies · doi:10.5555/2208461.2208466 · Cited by 72
Self-citationDEBAR: A scalable high-performance de-duplication storage system for backup and archiving
Yang, T., Jiang, H., Feng, D., Niu, Z., Zhou, K., & Wan, Y. (2010) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2010.5470468 · Cited by 71
Self-citationProviding High Reliability in a Minimum Redundancy Archival Storage System
Bhagwat, D., Pollack, K. T., Long, D. D. E., Schwarz, T. J. E., Miller, E. L., & Paris, J.-F. (2006) · Proceedings - International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems · doi:10.1109/mascots.2006.42 · Cited by 70
An analysis of compare-by-hash
Henson, V. (2003) · Workshop on Hot Topics in Operating Systems · Cited by 67
A Scalable Inline Cluster Deduplication Framework for Big Data Protection
Fu, Y., Jiang, H., & Xiao, N. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35170-9_18 · Cited by 64
Self-citationContent-dependent chunking for differential compression, the local maximum approach
Bjørner, N., Blass, A. R., & Gurevich, Y. G. (2009) · Journal of Computer and System Sciences · doi:10.1016/j.jcss.2009.06.004 · Cited by 63
SmartRE: an architecture for coordinated network-wide redundancy elimination
Anand, A., Sekar, V., & Akella, A. (2009) · ACM SIGCOMM Computer Communication Review · doi:10.1145/1594977.1592580 · Cited by 60
Cui, Y., Lai, Z., Wang, X., et al. (2015) · doi:10.1145/2789168.2790094 · Cited by 56
Xia, W., Jiang, H., Feng, D., & Tian, L. (2015) · IEEE Transactions on Computers · doi:10.1109/tc.2015.2456015 · Cited by 55
Self-citationCompare-by-hash: a reasoned analysis
Black, J. (2006) · USENIX Annual Technical Conference · Cited by 53
Xia, W., Jiang, H., Feng, D., & Tian, L. (2014) · DCC · doi:10.1109/dcc.2014.38 · Cited by 48
Self-citationLi, A., Shu, J., & Li, M. (2010) · Journal of Software · doi:10.3724/sp.j.1001.2010.03761 · Cited by 43
Estimation of deduplication ratios in large data sets
Harnik, D., Margalit, O., Naor, D., Sotnikov, D., & Vernik, G. (2012) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2012.6232381 · Cited by 39
A GPU accelerated storage system
Gharaibeh, A., Al-Kiswany, S., Gopalakrishnan, S., & Ripeanu, M. (2010) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/1851476.1851497 · Cited by 39
Asymmetric caching: improved network deduplication for mobile devices
Sanadhya, S., Sivakumar, R., Kim, K.-H., Congdon, P., Lakshmanan, S., & Singh, J. (2012) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/2348543.2348565 · Cited by 35
FAST: Near Real-Time Searchable Data Analytics for the Cloud
Hua, Y., Jiang, H., & Feng, D. (2014) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc.2014.67 · Cited by 31
Self-citationDesign of an exact data deduplication cluster
Kaiser, J., Meister, D., Brinkmann, A., & Effert, S. (2012) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2012.6232380 · Cited by 30
Improving the Resistance to Side-Channel Attacks on Cloud Storage Services
Heen, O., Neumann, C., Montalvo, L., & Defrance, S. (2012) · IFIP International Conference on New Technologies, Mobility and Security (NTMS) · doi:10.1109/ntms.2012.6208705 · Cited by 30
Modeling the Fault Tolerance Consequences of Deduplication
Rozier, E. W. D., Sanders, W. H., Zhou, P., Mandagere, N., Uttamchandani, S., & Yakushev, M. L. (2011) · IEEE Symposium on Reliable Distributed Systems (SRDS) · doi:10.1109/srds.2011.18 · Cited by 28
PeerDedupe: Insights into the Peer-Assisted Sampling Deduplication
Xing, Y., Li, Z.-C., & Dai, Y. (2010) · IEEE International Conference on Peer-to-Peer Computing (P2P) · doi:10.1109/p2p.2010.5570004 · Cited by 18
Secure deduplication of general computations
Tang, Y. S., & Yang, J. (2015) · USENIX Annual Technical Conference · Cited by 18
Cloud-scale Image Compression Through Content Deduplication
Perra, D., & Frahm, J. (2014) · British Machine Vision Conference (BMVC) · doi:10.5244/c.28.55 · Cited by 15
Cited by
Investigating the Adoption of Hybrid Encrypted Cloud Data Deduplication With Game Theory
Liang, X., Yan, Z., Deng, R. H., & Zheng, Q. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.3028685 · Cited by 18
Human-inspired Perspectives: A Survey on AI Long-term Memory
He, Z., Lin, W., Zheng, H., Zhang, F., Jones, M. W., Aitchison, L., Xu, X., Liu, M., Kristensson, P. O., & Shen, J. (2024) · arXiv · doi:10.48550/arxiv.2411.00489 · Cited by 3
“We review existing works to offer insights into the potential solutions to these challenges.”
Metadata
- Authors
- Xia, W., Jiang, H., Feng, D., Douglis, F., Shilane, P., Hua, Y., Fu, M., Zhang, Y., & Zhou, Y.
- Venue
- Proceedings of the IEEE
- Published
- 2016-09
- Volume
- 104
- Issue
- 9
- Pages
- 1681-1710
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W2475932436
- Cited by
- 318
- References
- 93
- FWCI
- 25.27
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.