The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Duplicate Record Detection: A Survey
Open accessAbstract
Often, in the real world, entities have two or more representations in databases. Duplicate records do not share a common key and/or they contain errors that make duplicate matching a difficult task. Errors are introduced as the result of transcription errors, incomplete information, lack of standard formats, or any combination of these factors. In this paper, we present a thorough analysis of the literature on duplicate record detection. We cover similarity metrics that are commonly used to detect similar field entries, and we present an extensive set of duplicate detection algorithms that can detect approximately duplicate records in a database. We also cover multiple techniques for improving the efficiency and scalability of approximate duplicate detection algorithms. We conclude with coverage of existing tools and with a brief discussion of the big open problems in the area
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Basic local alignment search tool
Altschul, S. F., Gish, W., Miller, W., Myers, E. W., & Lipman, D. J. (1990) · Journal of Molecular Biology · doi:10.1006/jmbi.1990.9999 · Cited by 98,669
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Needleman, S. B., & Wunsch, C. D. (1970) · Journal of Molecular Biology · doi:10.1016/0022-2836(70)90057-4 · Cited by 11,288
Identification of common molecular subsequences
Smith, T. F., & Waterman, M. S. (1981) · Journal of Molecular Biology · doi:10.1016/0022-2836(81)90087-5 · Cited by 9,940
Combining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
Maximum Likelihood from Incomplete Data via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1976) · Cited by 4,517
Text classification from labeled and unlabeled documents using EM
Nigam, K., Mccallum, A. K., Thrun, S., & Mitchell, T. (2000) · Machine Learning · doi:10.1023/a:1007692713085 · Cited by 2,736
A guided tour to approximate string matching
Navarro, G. (2001) · ACM Computing Surveys · doi:10.1145/375360.375365 · Cited by 2,561
Fellegi, I. P., & Sunter, A. B. (1969) · Journal of the American Statistical Association · doi:10.1080/01621459.1969.10501049 · Cited by 2,475
Improving generalization with active learning
Cohn, D., Atlas, L., & Ladner, R. (1994) · Machine Learning · doi:10.1007/bf00993277 · Cited by 1,591
Advances in Record-Linkage Methodology as Applied to Matching the 1985 Census of Tampa, Florida
Jaro, M. A. (1989) · Journal of the American Statistical Association · doi:10.1080/01621459.1989.10478785 · Cited by 1,401
Syntactic clustering of the Web
Broder, A. Z., Glassman, S. C., Manasse, M. S., & Zweig, G. (1997) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(97)00031-7 · Cited by 1,343
Techniques for automatically correcting words in text
Kukich, K. (1992) · ACM Computing Surveys · doi:10.1145/146370.146380 · Cited by 1,257
Efficient clustering of high-dimensional data sets with application to reference matching
McCallum, A., Nigam, K., & Ungar, L. H. (2000) · Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/347090.347123 · Cited by 1,096
Bansal, N., Blum, A., & Chawla, S. (2004) · Machine Learning · doi:10.1023/b:mach.0000033116.57574.95 · Cited by 1,075
Ristad, E. S., & Yianilos, P. N. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.682181 · Cited by 859
Real-world Data is Dirty: Data Cleansing and The Merge/Purge Problem
Hernández, M. A., & Stolfo, S. J. (1998) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009761603038 · Cited by 812
Newcombe, H. B., Kennedy, J. M., Axford, S. J., & James, A. P. (1959) · Science · doi:10.1126/science.130.3381.954 · Cited by 770
Fast text searching: allowing errors
Wu, S., & Manber, U. (1992) · Communications of the ACM · doi:10.1145/135239.135244 · Cited by 727
Interactive deduplication using active learning
Sarawagi, S., & Bhamidipaty, A. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775087 · Cited by 692
Research problems in data warehousing
Widom, J. (1995) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/221270.221319 · Cited by 642
Approximate string-matching with q-grams and maximal matches
Ukkonen, E. (1992) · Theoretical Computer Science · doi:10.1016/0304-3975(92)90143-4 · Cited by 605
A new approach to text searching
Baeza-Yates, R., & Gonnet, G. H. (1992) · Communications of the ACM · doi:10.1145/135239.135243 · Cited by 603
Adaptive name matching in information integration
Bilenko, M., Mooney, R., Cohen, W., Ravikumar, P., & Fienberg, S. (2003) · IEEE Intelligent Systems · doi:10.1109/mis.2003.1234765 · Cited by 469
Robust and efficient fuzzy match for online data cleaning
Chaudhuri, S., Ganjam, K., Ganti, V., & Motwani, R. (2003) · Proceedings of the 2003 ACM SIGMOD international conference on Management of data · doi:10.1145/872757.872796 · Cited by 457
Eliminating Fuzzy Duplicates in Data Warehouses
Ananthakrishna, R., Chaudhuri, S., & Ganti, V. (2002) · VLDB '02: Proceedings of the 28th International Conference on Very Large Databases · doi:10.1016/b978-155860869-6/50058-5 · Cited by 387
Cohen, W. W. (1998) · Proceedings of the 1998 ACM SIGMOD international conference on Management of data · doi:10.1145/276304.276323 · Cited by 379
Some biological sequence metrics
Waterman, M. S., Smith, T. F., & Beyer, W. A. (1976) · Advances in Mathematics · doi:10.1016/0001-8708(76)90202-4 · Cited by 358
Inverted files versus signature files for text indexing
Zobel, J., Moffat, A., & Ramamohanarao, K. (1998) · ACM Transactions on Database Systems · doi:10.1145/296854.277632 · Cited by 341
Fast parallel and serial approximate string matching
Landau, G. M., & Vishkin, U. (1989) · Journal of Algorithms · doi:10.1016/0196-6774(89)90010-2 · Cited by 328
Learning object identification rules for information integration
Tejada, S., Knoblock, C. A., & Minton, S. (2001) · Information Systems · doi:10.1016/s0306-4379(01)00042-4 · Cited by 310
TAILOR: a record linkage toolbox
Elfeky, M. G., Verykios, V. S., & Elmagarmid, A. K. (2003) · Proceedings 18th International Conference on Data Engineering · doi:10.1109/icde.2002.994694 · Cited by 289
Self-citationSutton, C., Rohanimanesh, K., & McCallum, A. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015422 · Cited by 284
Learning domain-independent string transformation weights for high accuracy object identification
Tejada, S., Knoblock, C. A., & Minton, S. (2002) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/775047.775099 · Cited by 257
Data integration using similarity joins and a word-based information representation language
Cohen, W. W. (2000) · ACM Transactions on Information Systems · doi:10.1145/352595.352598 · Cited by 223
Automatic segmentation of text into structured records
Borkar, V., Deshmukh, K., & Sarawagi, S. (2001) · Proceedings of the 2001 ACM SIGMOD international conference on Management of data · doi:10.1145/375663.375682 · Cited by 217
Static index pruning for information retrieval systems
Carmel, D., Cohen, D., Fagin, R., Farchi, E., Herscovici, M., Maarek, Y. S., & Soffer, A. (2001) · Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/383952.383958 · Cited by 207
Robust Identification of Fuzzy Duplicates
Chaudhuri, S., Ganti, V., & Motwani, R. (2005) · 21st International Conference on Data Engineering (ICDE'05) · doi:10.1109/icde.2005.125 · Cited by 170
Mining reference tables for automatic text segmentation
Agichtein, E., & Ganti, V. (2004) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1014052.1014058 · Cited by 152
Ullmann, J. R. (1977) · The Computer Journal · doi:10.1093/comjnl/20.2.141 · Cited by 134
Information Extraction: Distilling structured data from unstructured text
McCallum, A. (2005) · Queue · doi:10.1145/1105664.1105679 · Cited by 120
Merging the Results of Approximate Match Operations
Guha, S., Koudas, N., Marathe, A., & Srivastava, D. (2004) · Elsevier eBooks · doi:10.1016/b978-012088469-8.50057-7 · Cited by 111
Automating the approximate record-matching process
Verykios, V. S., Elmagarmid, A. K., & Houstis, E. N. (2000) · Information Sciences · doi:10.1016/s0020-0255(00)00013-x · Cited by 102
Self-citationLearning to match and cluster large high-dimensional data sets for data integration
Cohen, W. W., & Richman, J. (2002) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/775107.775116 · Cited by 83
A Model for Optimum Linkage of Records
Tepping, B. J. (1968) · Journal of the American Statistical Association · doi:10.1080/01621459.1968.10480930 · Cited by 67
Finding replicated Web collections
Cho, J. J., Shivakumar, N., & García-Molina, H. (2000) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/342009.335429 · Cited by 62
Mining database structure; or, how to build a data quality browser
Dasu, T., Johnson, T., Muthukrishnan, S., & Shkapenyuk, V. (2002) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/564716.564719 · Cited by 40
Entity matching in heterogeneous databases: a distance-based decision model
Dey, D., Sarkar, S., & De, P. (2002) · Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.1109/hicss.1998.649225 · Cited by 30
Cited by
Linked Data - The Story So Far
Bizer, C., Heath, T., & Berners-Lee, T. (2009) · International Journal on Semantic Web and Information Systems · doi:10.4018/jswis.2009081901 · Cited by 4,525
Linked Data: Evolving the Web into a Global Data Space
Heath, T., & Bizer, C. (2011) · Synthesis lectures on the semantic web · doi:10.2200/s00334ed1v01y201102wbe001 · Cited by 2,018
A comparative study of TF*IDF, LSI and multi-words for text classification
Zhang, W., Yoshida, T., & Tang, X. (2011) · Expert Systems with Applications · doi:10.1016/j.eswa.2010.08.066 · Cited by 676
A Survey of Link Prediction in Social Networks
Hasan, M. A., & Zaki, M. J. (2011) · Social Network Data Analytics · doi:10.1007/978-1-4419-8462-3_9 · Cited by 566
Systematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472
Metadata
- Authors
- Elmagarmid, A. K., Ipeirotis, P. G., & Verykios, V. S.
- Venue
- IEEE Transactions on Knowledge and Data Engineering
- Published
- 2007-01
- Volume
- 19
- Issue
- 1
- Pages
- 1-16
- ISSN
- 1041-4347 (ISSN-L)
- OpenAlex
- W3123518987
- Cited by
- 1,267
- References
- 49
- FWCI
- 31.66
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.