Identifying and Filtering Near-Duplicate Documents

On the resemblance and containment of documents

Broder, A. (2002) · Compression and Complexity of Sequences (SEQUENCES) · doi:10.1109/sequen.1997.666900 · Cited by 1,818

Syntactic clustering of the Web

Broder, A., Glassman, S. C., Manasse, M. S., & Zweig, G. (1997) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(97)00031-7 · Cited by 1,348

Min-Wise Independent Permutations

Broder, A., Charikar, M., FRIEZE, A. M., & Mitzenmacher, M. (2000) · Journal of Computer and System Sciences · doi:10.1006/jcss.1999.1690 · Cited by 812

Finding similar files in a large file system

Manber, U. (1994) · Cited by 597

Copy detection mechanisms for digital documents

Brin, S., Davis, J., & García-Molina, H. (1995) · Proceedings of the 1995 ACM SIGMOD international conference on Management of data - SIGMOD '95 · doi:10.1145/223784.223855 · Cited by 479

A technique for measuring the relative size and overlap of public Web search engines

Bharat, K. A., & Broder, A. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00127-5 · Cited by 392

Min-wise independent permutations (extended abstract)

Broder, A., Charikar, M., FRIEZE, A. M., & Mitzenmacher, M. (1998) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/276698.276781 · Cited by 364

Untitled work

(1995) · Cited by 296

Some applications of Rabin’s fingerprinting method

Broder, A. (1993) · Springer eBooks · doi:10.1007/978-1-4613-9323-8_11 · Cited by 204

Building a scalable and accurate copy detection mechanism

Shivakumar, N., & García-Molina, H. (1996) · ACM international conference on Digital libraries · doi:10.1145/226931.226961 · Cited by 161

The Search of New Issues in the Detection of Near-duplicated Documents

Naderi, H., Salehpour, N., Farokhi, M. N., & Hosseini, B. (2014) · doi:10.6084/m9.figshare.954921.v1 · Cited by 1

Evaluation of an automatic process for specialized web corpora collection and term extraction for Basque

Hernaiz, A. G., Azkarate, I. L., Irigoyen, E. P., Urizar, X. S., & Vicente, I. S. (2010) · Dialnet (Universidad de la Rioja) · Cited by 1

Duplicate text detection based on LCS algorithm

Yu, J., Li, M., & Zhang, D. (2016) · IEEE Information Technology and Mechatronics Engineering Conference (ITOEC) · doi:10.2991/itoec-16.2016.2 · Cited by 1

Mining Health Data using Weighted Approach

Priyanga, P., & Naveen, N. C. (2016) · Communications on Applied Electronics · doi:10.5120/cae2016652381 · Cited by 1

The perils of cross-silo deduplication: Trading user security for provider storage efficiency

Rozier, E. W. D. (2013) · IEEE International Workshop on Information Forensics and Security · doi:10.1109/wifs.2013.6707799 · Cited by 1

Aggregating sentence-level features for Chinese near-duplicate document detection

Liang, Y. C., Tao, Y., Feng, N., Wan, Z., Xu, F., Jiang, X., & Gao, S. (2017) · IEEE International Conference on Networking, Sensing and Control (ICNSC) · doi:10.1109/icnsc.2017.8000087 · Cited by 1

Co-relation of Image and Text by Using Semantic Cross-Media Hashing

Prameela, T., & Reddy, B. R. (2017) · doi:10.22161/ijaers/nctet.2017.53 · Cited by 1

DAMPAK SISTIM PENENTUAN KOSA KATA INDEKS PADA KARYA MONOGRAF TERHADAP TEMU BALIK INFORMASI PEMUSTAKA DI DINAS PERPUSTAKAAN DAN ARSIP KOTA JAMBI

Harys, K. (2017) · Baitul Ulum Jurnal Ilmu Perpustakaan dan Informasi · doi:10.30631/baitululum.v1i0.19 · Cited by 1

Графи видимості — інструмент мережевого аналізу рядів вимірів

Snarskii, A. A., & Lande, D. (2013) · Реєстрація зберігання і обробка даних · doi:10.35681/1560-9189.2013.15.2.103376 · Cited by 1

An Approach to Mining Scholars' Research Interests based on Academic Papers and Citation Networks

He, Z., Zhou, G., Lu, J., & Zheng, J. (2022) · IEEE Joint International Information Technology and Artificial Intelligence Conference (ITAIC) · doi:10.1109/itaic54216.2022.9836864 · Cited by 1

Identifying Near-Duplicate Pairs in Exams: Similarity Detection Using Kshingling, Minhashing, and LSH

Rhezzali, N. E., Hilal, I., & Hnida, M. (2023) · International Conference on Intelligent Systems: Theories and Applications · doi:10.1109/sita60746.2023.10373698 · Cited by 2

Near-Duplicate Detection for Online-Shops Owners: An FCA-Based Approach

Ignatov, D. I., Konstantiov, A. V., & Chubis, Y. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-36973-5_69 · Cited by 1

Improving Near-Duplicate Detection in Multi-Layered Collaborative Requirements Engineering Discussions Through Discussion Clustering

Sillaber, C., & Breu, R. (2013) · Springer proceedings in complexity · doi:10.1007/978-94-007-7287-8_20 · Cited by 1

Content sharing in information storage and retrieval system using tree representation of documents

Sharma, D., & Jain, S. (2014) · doi:10.1109/csibig.2014.7056941 · Cited by 1

A pruning strategy to improve pairwise comparison-based near-duplicate detection

Hassanian-esfahani, R., & Shayegan, M. J. (2019) · Knowledge and Information Systems · doi:10.1007/s10115-018-1299-2 · Cited by 1

RNACache: A scalable approach to rapid transcriptomic read mapping using locality sensitive hashing

Cascitti, J., Niebler, S., Müller, A., & Schmidt, B. (2022) · Journal of Computational Science · doi:10.1016/j.jocs.2022.101572 · Cited by 1

An extended version of sectional MinHash method for near-duplicate detection

Shayegan, M. J., & Faizollahi-Samarin, M. (2022) · The Journal of Supercomputing · doi:10.1007/s11227-022-04447-x · Cited by 1

Secure Deduplication

Feng, D. (2022) · Springer eBooks · doi:10.1007/978-981-19-0112-6_6 · Cited by 1

Post-deduplication Delta Compression Schemes

Feng, D. (2022) · Springer eBooks · doi:10.1007/978-981-19-0112-6_7 · Cited by 1

A Model of Uncertainty for Near-Duplicates in Document Reference Networks

Heß, C., & de Rougemont, M. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74851-9_40 · Cited by 1

Enhancing Duplicate Collection Detection Through Replica Boundary Discovery

Zhang, Z., Jia, W., & Li, X. (2006) · Lecture notes in computer science · doi:10.1007/11731139_42 · Cited by 1

Sketch-Based Estimation of Subpopulation-WeightPreprint

Cohen, E., & Kaplan, H. Y. (2008) · arXiv (Cornell University) · doi:10.48550/arxiv.0802.3448 · Cited by 1

Surveying Data for Patchy Structure

Pearson, R. K. (2005) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972757.3 · Cited by 1

Decomposing file data into discernible items

Policroniades, C. (2021) · CL Technical Reports · doi:10.48456/tr-672 · Cited by 1

Understanding the fault-tolerance properties of large-scale storage systems

Sanders, W. H., & Rozier, E. W. D. (2011) · Cited by 1

Differentially Private Weighted SamplingPreprint

Cohen, E., Geri, O., Sarlós, T., & Stemmer, U. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13048 · Cited by 1

Variance Competitiveness for Monotone Estimation: Tightening the BoundsPreprint

Cohen, E. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6490 · Cited by 1

Research on Indexing Page Collection Selection Method for Search Engine

Ru, L., Li, Z., Wu, Y., & Ma, S. (2013) · Springer proceedings in complexity · doi:10.1007/978-1-4614-6880-6_30

Toward Building Practical Privacy Enhancing Technologies

Lin, Z. (2012) · University of Minnesota Digital Conservancy (University of Minnesota)

Порождение кластеров документов-дубликатов: подход, основанный на поиске частых замкнутых множеств признаков

Kuznetsov, S. O., Ignatov, D. I., Объедков, С. А., & Самохин, М. В. (2005) · Electronic scientific archive of UrFU (Ural Federal University)

Compactified Horizontal Visibility Graph for the Language NetworkPreprint

Lande, D., & Snarskii, A. A. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1302.4619

A Webpage Deletion Algorithm Based on Hierarchical Filtering

Chen, X., Wang, W., Man, D., & Xuan, S. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33469-6_68

Exploiting the Computational Power of Ternary Content Addressable Memory

Tirdad, K. (2011) · UWSpace (University of Waterloo)

Parallelized Near-Duplicate Document Detection Algorithm for Large Scale Chinese Web Pages

Wei, Y., Wang, S., Yuan, C., & Huang, Y. (2012) · International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT) · doi:10.1109/pdcat.2012.108

A novel approach for leveraging co-occurrence to improve the false positive error in signature files

Ghodsnia, P., Tirdad, K., Munro, J. I., & López-Ortíz, A. (2012) · Journal of Discrete Algorithms · doi:10.1016/j.jda.2012.10.006

Adaptive CDCLZW algorithm for data compression

Liu, X., Liao, J., & Shen, Q. (2013) · IEEE International Conference on Broadband Network and Multimedia Technology (IC-BNMT) · doi:10.1109/icbnmt.2013.6823919

TurboSockets: Democratizing Distributed Deduplication

Salada, J. P., & Barreto, J. (2013) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom.2013.154

Detecting and Clustering Similar Results of Search Engine by Exploiting Web Page's Contents

Gao, K., & Wu, H. (2008) · International Conference on Wireless Communications, Networking and Mobile Computing · doi:10.1109/wicom.2008.2548

CONTENT ADDRESSABLE DATA MANAGEMENT

Nath, P. (2007)

Optimizing information extraction programs over evolving text

Doan, A., Ramakrishnan, R., & Chen, F. (2010)

Information Retrieval for E-Discovery (( DRAFT: $Revision: 461 $ $Date: 2013-05-16 10:37:00 -0400 (Thu, 16 May 2013) $))

Oard, D. W., & Webber, W. (2013)

Within-Language Information Retrieval

Peters, C., Braschler, M., & Clough, P. D. (2011) · Springer eBooks · doi:10.1007/978-3-642-23008-0_2

Fuzzy Fingerprint Method for Detection of Sensitive Data Exposure

Ulahannanl, S., & Jose, R. (2016) · International Journal on Cybernetics & Informatics · doi:10.5121/ijci.2016.5207

Efficient Similarity Search in Dynamic Data StreamsPreprint

Bury, M., Schwiegelshohn, C., & Sorella, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.03949

Detection for Trusted Content Delivery Networks Traffic by Pattern-based Content Leakage

Suryawanshi, M., & Patil, S. (2016)

A nugget-based test collection construction paradigm

Rajput, S. (2012) · doi:10.17760/d20002774

Near Duplicate Document Detection using Document Image

Uwamahoro, G., Zhang, Z., Mtafya, A. R., Li, W., & Jun, L. (2016) · International Journal of Multimedia and Ubiquitous Engineering · doi:10.14257/ijmue.2016.11.7.17

Clustering of the External WEB Environment of Universities Using a Modified LSH Algorithm

Корелин, В. Н., Blekanov, I. S., & Sergeev, S. S. (2015) · St Petersburg State Polytechnical University Journal Computer Science Telecommunications and Control Systems · doi:10.5862/jcstcs.229.8

Identify Similar Research Papers Using Locality Sensitive Hashing

Gupta, D., & Batra, S. (2016)

Dissertation submitted to obtain the Master of Science Degree in Communication Networks Engineering

Salada, J. P. (2013)

Malware Mitigation and Remediation Strategies

Passerini, E., Bruschi, D., Dottorato, C., & Damiani, E. (2009)

A comprehensive method for automating test collection creation and evaluation for retrieval and summarization systems

Ekstrand-Abueg, M. (2016) · doi:10.17760/d20237565

depuis l'arabe

Gahbiche, S., Sud, P., & Xi, P. (2013)

Detection Of Duplicate And Near-Duplicate Content For Web Crawlers

Khan, H. H., & Ali, H. M. (2015) · Journal of Independent Studies and Research - Computing · doi:10.31645/jisrc/(2015).13.2.0002

Evaluation of intra-set clustering techniques for redundant social media content

Jubinville, J. (2018) · UVic’s Research and Learning Repository (University of Victoria)

A Heterogeneous High Dimensional Approximate Nearest Neighbor AlgorithmPreprint

Dubiner, M. (2008) · arXiv (Cornell University) · doi:10.48550/arxiv.0810.4188

Algorithms for Similarity Search and PseudorandomnessPreprint

Christiani, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.09430

Scalable Analysis of Open Data Graphs

Stoica, A., Valdron, M., & Pu, K. Q. (2019) · IEEE International Conference on Information Reuse and Integration (IRI) · doi:10.1109/iri.2019.00059

Understanding Data Similarity in Large-Scale Scientific Datasets

Linton, P., Melodia, W., Lazar, A., Agarwal, D., Bianchi, L., Ghoshal, D., Junior, G. Z. P., Ramakrishnan, L., & Wu, K. J. (2019) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata47090.2019.9006471

Alignment-free Phylogeny Reconstruction Based On Quartet Trees

Dencker, T. (2020) · eDiss (Georg-August-Universität Göttingen) · doi:10.53846/goediss-7916

Using fingerprints based on the sequence of a set of selected words in a document for 1-to-n similarity analysis

Gupta, G., & Singla, A. (2022) · Figshare · doi:10.26180/20365410

MODEL FOR DETERMINING THE NOVELTY OF A REQUEST FOR PROJECT FINANCING VIA TERMINOLOGICAL ANALYSIS METHODS

Sirotkin, A. V., & Константинович, К. В. (2020) · Современные наукоемкие технологии (Modern High Technologies) · doi:10.17513/snt.38003

Identification automatique des sources des notices zoologiques du Speculum naturale de Vincent de Beauvais

Cuvelier, É., de Valeriola, S., & Engelbeen, C. (2020) · Revue Ouverte d Intelligence Artificielle · doi:10.5802/roia.2

Semantic-Based Duplicate Web Page Detection

Sheela, A. C. S., & Jayakumar, C. (2021) · Lecture notes in electrical engineering · doi:10.1007/978-981-15-8685-9_56

Incentives for Item Duplication under Fair Ranking PoliciesPreprint

Nunzio, G. M. D., Fabris, A., Silvello, G., & Susto, G. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15683

Plagiarism detection by similarity join

Schellenberger, R. (2009) · Research Repository (Delft University of Technology)

Feature Hashing with Insertion and Deletion of Features

Pratap, R., Bhardwaj, S., Sarode, H. S., & Kulkarni, R. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9672011

Implementation of Automated Industry-Specific Identification of a Grant Application Based on Tokenization with a Predetermined Weight

Sirotkin, A. V., & Константинович, К. В. (2020) · International Theoretical and Practical Conference "The Crossroads of the North and the East" · doi:10.32743/nesu.cross.2020.207-215

Detecting Document Versions and Their Ordering in a Collection

Modani, N., Maurya, A., Verma, G., Nair, I., Patil, V., & Kanfade, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91560-5_30

Investigation Framework for Cloud Forensics using Dynamic Genetic-based Clustering

AlKhanafseh, M., Qatawneh, M., & Almobaideen, W. A. (2022) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2022.01301100

Coordinated Sampling

Cohen, E. (2014) · Encyclopedia of Algorithms · doi:10.1007/978-3-642-27848-8_576-1

A Backup Service System with Log-based De-duplication and Consistency Guarantee

Wang, H., Zhou, K., & Suo, N. (2011) · Energy Procedia · doi:10.1016/j.egypro.2011.12.593

Collaborative Data Gathering in Wireless Sensor Networks Using Measurement Co-Occurrence

Kalpakis, K., & Tang, S. (2007) · International Conference on Sensor Technologies and Applications · doi:10.1109/sensorcomm.2007.4394985

A Machine Learning Approach to Detect Differential Treatment of Anonymous Users

Wagner, I. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17146-8_1

Deep Packet Inspection at Scale: Search Optimization Through Locality-Sensitive Hashing

Kapoor, M., Krishnan, S., & Moyer, T. (2022) · IEEE International Symposium on Network Computing and Applications (NCA) · doi:10.1109/nca57778.2022.10013504

A multi‐feature‐based intelligent redundancy elimination scheme for cloud‐assisted health systems

Xiao, L., Zou, B., Kui, X., Zhu, C., Zhang, W., Yang, X., & Zhang, B. (2023) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.12211

Enabling Encrypted Delta Compression for Outsourced Storage Systems via Preserving Similarity

Gan, C., Hu, Y., Zhao, L., Zhao, X., Gong, P., Zhang, W., Wang, L., & Feng, D. (2023) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd58817.2023.00043

From Reinvention to Reuse: An Empirical Example Study on Technical Debt Dataset

Rantala, L., Mäntylä, M., & Sridharan, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78386-9_8

ERD: AVX-512-based Enhancement of Resemblance Detection for Post-Deduplication Delta Compression

Chen, T., Liu, H., Fang, W. S., Yan, W., Mo, N., & Liu, X. (2024) · IEEE International Conference on Networking, Architecture and Storage (NAS) · doi:10.1109/nas63802.2024.10781356

Secure Fuzzy Deduplication: Definitions and Constructions

Chakrabarti, A., Gupta, S., Kanukurthi, B., & Shankar, G. (2025) · doi:10.1145/3733812.3765537

Novelty Detection in Knowledge Base Acceleration

Andresen, E. W. (2013) · BIBSYS Brage (BIBSYS (Norway))

Additional file 4 of STAT: a fast, scalable, MinHash-based k-mer tool to assess Sequence Read Archive next-generation sequence submissions

Katz, K., Shutov, O., Lapoint, R. T., Kimelman, M., Brister, J. R., & O’Sullivan, C. (2021) · Figshare · doi:10.6084/m9.figshare.16650491

Hybrid-Rewrite: A Rewriting Framework for Hybrid Deduplication and Delta Compression

Li, Q., Jiang, H. M., Zhang, Y., Xu, Z., Wu, J., & Lu, P. (2025) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd65941.2025.00071

Enhancing LLM to Decompile Optimized PTX to Readable CUDA for Tensor Programs

Sun, X., Tang, F., Zhang, Y., Shen, H., Song, C., & Zhang, D. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00185

Language models reveal evidence gaps in variants of uncertain significancePreprint

Li, W., Bhat, V., Yu, T., Lebo, M. S., Žitnik, M., & Cassa, C. A. (2026) · medRxiv · doi:10.64898/2026.02.28.26347206

Index Intersection for High-Dimensional Range Queries

Berens, M., & Teubner, J. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3785297.3785315

ModSolAgent: Automated Finite Element Code Generation for Abaqus via LLM-Based Agent

Li, Z., Ge, H., Tang, G., & Liu, Y. (2026) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2026.3669495

Amélioration des systèmes de traduction par analyse linguistique et thématique : application à la traduction depuis l'arabe

Gahbiche, S. (2013) · doi:10.70675/e8e9227dzeffez4b56za431zacbd1d2ccaea · Cited by 2

Qualité et réutilisabilité des données dans la conception de services innovants

Foulonneau, M. (2023) · theses.fr (ABES) · doi:10.70675/2abf9dffz9739z4248zb127z8b07979870d6

DuCodeMark: Dual-Purpose Code Dataset Watermarking via Style-Aware Watermark-Poison Design

Chen, Y., Xiao, Y., Fang, C., Chen, Z., & Xu, B. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797076

CodeCAD: A Parametric CAD Dataset for Programmatic 3D Model Generation

Fresacher, D., & Diepold, K. J. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817592

The Enduring Potential of Locality: Efficient Routing for Fine-Grained Cluster Deduplication

Deng, C., Zou, X., Shilane, P., Tan, H., Li, S., & Xia, W. (2026) · International Workshop on Quality of Service · doi:10.1109/iwqos70441.2026.11661042

Amplification in Locality-Sensitive Hashing for Webshop Product Deduplication

Bos, N., Wadolowski, A., & Frăsincar, F. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3729858

Efficient Similarity Detection for Delta Compression

Jin, X., & Liu, H. (2026) · Computers · doi:10.3390/computers15090607

SimPattern: A Novel Byte-Pattern Matching based Chunk Similarity Detection for Post-Deduplication

Cai, Z., Ye, X., Tian, W., Li, R., Wang, H., Lu, J., He, Z., Xu, Z., & Ingle, P. Y. (2026) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3832810.3832926

FakenewsBR: A Unified Portuguese Misinformation Corpus for Reproducible Data Integration Research

Mariano, K. D. P., Almada, F. L. N., Dutra, M. A., Monteiro, V. E. D. S., Gomes, J. R. S., Filho, A. R. G., & Soares, A. D. S. (2026) · doi:10.5753/dsw.2026.249457

Cross-Backup First: Prioritizing Historical Base Chunks for Delta Compression in Deduplicated Backup Systems

Li, C., Xie, C., Liu, J., & Zhang, Y. (2026) · Electronics · doi:10.3390/electronics15204712

Filtering near-duplicate documents

Broder, A. (1998)

Near duplicate detection in large document collections

Benčík, D. (2007) · Digital Repository (National Repository of Grey Literature)

Estimating the Number of Near-Duplicate Document Pairs for Massive Data Sets using Small Space

Deng, F., & Rafiei, D. (2008)

Concept based algorithm of dealing near-replicas of documents on the Web

Li, Z. (2006) · Journal of Computer Applications · Cited by 1

Near Duplicate Document Detection using Document Image

Uwamahoro, G., Zhang, Z., Mtafya, A. R., Li, W., & Jun, L. (2016) · International Journal of Multimedia and Ubiquitous Engineering · doi:10.14257/ijmue.2016.11.7.17